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EXECUTIVE  SUMMARY 


In  1990,  the  Office  of  the  Assistant  Secretary  of  Defense  for  Production  and  Logistics  introduced  a 
guide,  SD-2,  for  buying  Non-Developmental  Item  (NDI)  systems.  The  guide  was  updated  in  1995 
as  DoD  5000.37-H,  “Buying  Commercial  and  Nondevelopmental  Items:  A  Handbook,”  and  several 
companion  “SD”  guides  have  been  issued.  DoD  policy  recognizes  that  NDI  acquisition  represents  a 
cost-effective  approach  for  meeting  a  variety  of  system  requirements.  The  guide  recognizes  that 
NDI  acquisition  procedures  are  not  new  or  significantly  different  from  other  types  of  acquisitions; 
nevertheless,  the  guide  also  recognizes  that  certain  issues  must  be  resolved  in  order  to  successfully 
meet  military  requirements  without  compromise  while  achieving  the  benefits  promised  by  NDI. 

SD-2  states  in  its  foreword,  “The  Department  of  Defense  must  explore  and  implement  NDI  solutions 
which  provide  best  value  in  terms  of  life-cycle  cost,  system  capability,  supportability,  and  quality.” 
This  Center  has  been  involved  in  acquisition  research  since  the  early  1970s  and  has  explored  the 
promises  and  pitfalls  of  NDI  acquisitions.  Much  of  the  early  work  was  documented  in  the  Center’s 
Technical  Document  108,  “Project  Management  and  Systems  Engineering  Guide,”  first  published  in 
1 977 .  Since  that  time,  the  policies  of  acquiring  NDI  systems  have  produced  additional  opportunities 
to  study  NDI  promises  and  pitfalls  on  a  greater  variety  of  projects.  Many  of  the  pitfalls  have  been 
found  to  be  related  to  the  failure  to  adopt  a  systems  engineering  approach  to  system  life-cycle  man¬ 
agement  rather  than  characteristics  of  NDI  acquisitions. 

This  document  summarizes  the  best  practices  and  lessons  learned  in  several  decades  of  acquiring 
NDI  systems.  These  processes  have  been  codified  into  a  tailorable  process  and  several  subprocesses. 
This  process  has  been  reconciled  with  the  Center’s  systems  engineering  processes  and  practices  to 
ensure  its  practical  application.  The  process  also  introduces  the  concept  of  a  system  life-cycle  man¬ 
agement  agent  responsible  for  overseeing  the  process  application  throughout  the  system  life  cycle. 
The  heavy  application  of  NDI  to  system  acquisitions  raises  a  variety  of  very  significant  issues  that 
must  be  resolved  across  traditional  organizational  boundaries  and  that  require  the  coordination  of  a 
variety  of  engineering  expertise;  the  system  life-cycle  management  concept  provides  for  this  techni¬ 
cal  coordination  and  ensures  that  appropriate  expertise  is  applied  to  the  resolution  of  these  issues. 
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WHAT  IS  NDI? 


NDI  stands  for  Non-Developmental  Item.  This  is  a  deceptively  simple  definition  because  it 
sounds  like  a  single  thing  or  set  of  things — one  set  of  problems  having  a  single  solution.  It  is  not. 
NDI  really  defines  an  entire  spectrum  of  products  that  have  dramatically  different  properties.  As  a 
result,  no  single  set  of  rules  guides  the  NDI  buyer  for  all  circumstances.  Nevertheless,  NDI  poses 
issues  that  should  be  addressed  differently  from  the  “build  it  from  scratch”  mode  of  acquisition.  This 
guide  addresses  the  issues  encountered  during  system  acquisition  from  the  NDI  perspective. 

NDI  includes  the  following  classes  of  NDI,  each  defined  by  its  own  unique  characteristics  and 
challenges. 

OFF-THE-SHELF  CLASSES  OF  NDI 

Military  Ojf-the-Shelf  (MILOTS) — existing  products  specifically  and  uniquely  designed  for  one 
military  application  being  applied  to  a  new  military  application. 

Government  Ojf-the-Shelf  (GOTS) — existing  products  designed  for  government  applications  where 
the  government  owns  the  design.  (GOTS  is  inclusive  of  all  MILOTS,  but  also  includes  many  other 
products  in  both  hardware  and  software.) 

Foreign  Military  Equipment  (FME) — equipment  equivalent  to  MILOTS  but  designed  for  foreign 
military  applications  where  the  design  is  owned  by  a  foreign  government,  including  NATO  standard 
items. 

Commercial  Off-the-Shelf  (COTS) — ^products  offered  in  the  open  market;  the  design  is  owned  and 
controlled  by  the  supplier.  See  the  section  on  COTS  for  the  definition  of  different  grades  of  COTS. 

Rugged  (or  Ruggedized)  Off-the-Shelf  (ROTS) — COTS  where  the  supplier  has  modified  the  design 
to  meet  the  more  stringent  environments  or  special  characteristics  often  associated  with  military 
applications.  See  the  section  on  COTS  for  the  definition  of  different  grades  of  COTS/ROTS. 

Foreign  Commercial  Off-the-Shelf  (FCOTS)— COTS  where  the  design  is  controlled  by  an  supplier 
that  is  not  domestic  to  the  United  States. 

NewCOTS — any  form  of  COTS  or  ROTS  that  does  not  have  at  least  1  year  of  field  experience  or 
support  of  high-volume/continuous  production.  An  item  not  in  continuous  production  must  have  at 
least  5  years  of  field  experience  to  not  be  considered  NewCOTS. 

Non-Developmental  Software  (NDS) — any  software  configuration  item  that  is  used  without  modifi¬ 
cation  and  where  the  design  control  agent  of  the  software  is  not  the  System  Design  Agent  for  the 
application  or  system  being  acquired.  [NOTE:  NDS  can  fall  into  any  of  the  classes  of  off-the-shelf 
products  above.] 

Reuse  Software — any  software  that  is  used  without  modification  and  where  the  design  control 
agent  of  the  software  is  also  the  System  Design  Agent  for  the  application  or  system  being  acquired. 
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NON-OFF-THE-SHELF  CLASSES  OF  NDI 

Modified  Off-the-Shelf  (MOTS) — an  off-the-shelf  item  from  any  of  the  classes  above  that  must  be 
modified  or  adapted  through  changes  to  the  target  application. 

Minor  Modification  a  modification  where  the  scope  of  the  change  does  not  exceed  2  percent  of 
the  total  product  complexity  (in  terms  of  parts  count,  entity  count,  source  lines  of  code,  or  other 
sirmlar  uniform  measure).  (Final  percent  changes  in  the  design  as  high  as  5  percent  are  some¬ 
times  accepted.) 

Major  Modification  a  modification  where  the  scope  of  change  exceeds  that  of  a  minor  modifi¬ 
cation  but  remains  less  than  a  30-percent  change  in  total  product  complexity. 

Integrated  NDI— x\\Q  interfacing  of  pre-existing  hardware  and/or  software  configuration  items  to 
create  a  new  system  design. 


Integrated  Modification  of  NDI — integrated  NDI  that  also  involves  the  modification  of  one  or 
more  of  the  configuration  items  or  that  requires  the  development  of  one  or  more  configuration  items 
to  effect  the  integration  of  the  other  items.  The  definitions  of  minor  and  major  modification  above 
also  apply.  In  addition,  newly  developed  configuration  items  cannot  in  total  exceed  10  percent  of  the 
total  system  complexity  as  measured  by  a  uniform  metric.  No  more  than  20  percent  of  the  system 
functionality  should  be  allocated  to  newly  designed  configuration  items.  System  designs  exceeding 
these  thresholds  should  be  considered  new  developments  rather  than  NDI  acquisitions. 

Two  important  NDI  issues  become  apparent  in  these  class  definitions.  The  first  issue  is  design 
ownership.  Design  ownership  requires  the  control  of  the  design  specifications  and  configuration 
management  of  the  product  baseline  (the  elements  needed  to  produce  and  reproduce  the  product  and 
its  supported  components).  All  of  the  off-the-shelf  forms  of  NDI  involve  design  ownership  by  an 
agent  other  than  the  .S>  stem  Design  Agent.  The  other  forms  of  NDI  involve  design  ownership  of 
modification  designs  h\  the  System  Design  Agent.  The  second  issue  is  product  maturity.  All  COTS 
Other  than  New  COTS  can  be  considered  mature.  NewCOTS  and  the  non-_OTS  forms  of  NDI  have 
significant  system  components  that  are  immature.  System  maturity  impacts  quality,  reliability,  and 
supportability  issues.  These  two  issues,  design  ownership  and  maturity,  form  the  basis  of  assump¬ 
tions  that  are  applied  in  the  various  guidelines. 

The  guidance  in  this  document  has  been  developed  and  validated  through  many  years  of  practical 
experience  in  buying  NDI.  NDI  has  been  successfully  applied  to  many  unique  and  specialized  mili¬ 
tary  applications.  A  common  misconception  is  that  a  unique  military  application  must  always  result 
in  a  unique  militarx  design.  This  is  often  not  the  case.  Most  unique  applications  still  have  substan¬ 
tial  elements  of  functionality  in  common  with  a  broad  base  of  applications.  As  a  result,  it  is  almost 
always  possible  to  use  some  form  of  NDI.  In  fact,  it  is  very  rare  to  base  system  functionality  on  a 
newly  invented  technical  component;  at  some  level,  even  systems  built  from  scratch  are  built  of  NDI 
(such  as  integrated  circuits,  power  supplies,  capacitors,  connectors)  integrated  in  a  new  way.  The 
distinction  between  the  class  of  NDI  that  involves  extensive  integration  and  new  design  is  effectively 
the  same  as  the  distinction  between  system  design  and  design  engineering.  To  the  degree  that  this 
boundary  can  be  very  fuzzy,  so  too  the  distinction  between  new  design  and  integrated  NDI.  Devel¬ 
opmental  acquisitions  do,  however,  share  distinguishing  features  that  include  the  ability  to  control 
design  features  and  supporting  documentation  down  to  and  below  the  level  of  system  complexity  at 
which  the  system  is  supported.  The  development  system  is  not  mature  because  it  is  a  new  design. 

NDI  acquisitions  do  not  have  this  degree  of  design  control,  but  the  resulting  systems  enjoy  a  high 
degree  of  mamrity. 
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DEFINITIONS  OF  COTS  AND  NDI  IN  THE  FEDERAL  ACQUISITION  REGULATIONS  (FAR) 

The  classes  of  NDI  defined  above  are  driven  by  engineering  considerations.  The  FAR,  as  a  result 
of  the  Federal  Acquisition  Streamlining  Act  (FAS A)  of  1994,  distinguishes  between  a  “Commercial 
Item”  (COTS,  ROTS,  FOOTS,  NewCOTS,  and  the  minor  modifications  of  these  classes)  and  a 
“Non-Developmental  Item”  (NDI  including  MILOTS,  FME,  and  the  modifications  of  these  classes). 
GOTS  is  not  covered  in  the  FAR.  The  Commercial  Item  definition  is  as  follows: 

(a)  Any  item,  other  than  real  property,  that  is  of  a  type  customarily  used  for  nongovernmental  pur¬ 
poses  and  that 

(1)  Has  been  sold,  leased,  or  licensed  to  the  general  public;  or 

(2)  Has  been  offered  for  sale,  lease,  or  license  to  the  general  public; 

(b)  Any  item  that  evolved  from  an  item  described  in  paragraph  (a)  of  this  definition  through 
advances  in  technology  or  performance  and  that  is  not  yet  available  in  the  commercial  market¬ 
place,  but  will  be  available  in  the  commercial  marketplace  in  time  to  satisfy  the  delivery 
requirements  under  a  government  solicitation; 

(c)  Any  item  that  would  satisfy  a  criterion  expressed  in  paragraphs  (a)  or  (b)  of  this  definition,  but 
for 

(1)  Modification  of  a  type  customarily  available  in  the  commercial  marketplace;  or 

(2)  Minor  modifications  of  a  type  not  customarily  available  in  the  commercial  marketplace 
made  to  meet  Federal  Government  requirements; 

(d)  Any  combination  of  items  meeting  the  requirements  of  paragraphs  (a),  (b),  (c),  or  (e)  of  this 
definition  that  are  of  a  type  customarily  combined  and  sold  in  combination  to  the  general 
public; 

(e)  Installation  services,  maintenance  services,  repair  services,  training  services,  and  other  services 
if  such  services  are  procured  for  support  of  an  item  referred  to  in  paragraphs  (a),  (b),  (c),  or  (d) 
of  this  definition,  and  if  the  source  of  such  services; 

(1)  Offers  such  services  to  the  general  public  and  the  Federal  Government  contempora¬ 
neously  and  under  similar  terms  and  conditions;  and 

(2)  Offers  to  use  the  same  work  force  for  providing  the  Federal  Government  with  such  ser¬ 
vices  as  the  source  uses  for  providing  such  services  to  the  general  public; 

(f)  Services  of  a  type  offered  and  sold  competitively  in  substantial  quantities  in  the  commercial 
marketplace  based  on  established  catalog  or  market  prices  for  specific  tasks  performed  under 
standard  commercial  terms  and  conditions.  This  does  not  include  services  that  are  sold  based 
on  hourly  rates  without  an  established  catalog  or  market  price  for  a  specific  service  performed; 

(g)  Any  item,  combination  of  items,  or  service  referred  to  in  paragraphs  (a)  through  (f),  notwith¬ 
standing  the  fact  that  the  item,  combination  of  items  or  service  is  transferred  between  or 
among  separate  divisions,  subsidiaries,  or  affiliates  of  a  contractor; 

(h)  The  procuring  agency  determines  that  the  item  was  developed  exclusively  at  private  expense 
and  sold  in  substantial  quantities,  on  a  competitive  basis,  to  multiple  state  and  local  govern¬ 
ments. 
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The  FAR  defines  a  Non-Developmental  Item  as  follows: 

(a)  Any  previously  developed  item  of  supply  used  exclusively  for  governmental  purposes  by  a 
Federal  Agency,  a  state  or  local  government,  or  a  foreign  government  with  which  the  United 
States  has  a  mutual  defense  cooperation  agreement; 

(b)  Any  item  described  in  paragraph  (a)  of  this  definition  that  requires  only  minor  modification  or 
modifications  of  a  type  customarily  available  in  the  commercial  marketplace  in  order  to  meet 
the  requirements  of  the  procuring  department  or  agency;  or 

(c)  Any  item  of  supply  being  produced  that  does  not  meet  the  requirements  of  paragraph  (a)  or 
(b),  solely  because  the  item  is  not  yet  in  use. 

The  FAR  separates  commercial  items  and  NDI  such  that  a  commercial  item  is  not  NDI;  although 
some  NDI  may  be  termed  a  commercial  item.  This  is  a  legal  distinction  made  for  purposes  of  the 
acquisition  procedure  issues,  in  contrast  to  engineering  system  design  and  management  issues.  Both 
the  FAR  and  this  document  recognize  the  substantial  differences  between  existing  items  and  new 
developmental  items  and  their  respective  acquisitions.  Both  approaches  recognize  that  the  acquiring 
agency  should  not  be  concerned  with  the  internal  design  of  the  item — externally  specified  require-  ° 
ments  are  the  sole  criteria  for  make  a  source  selection. 

GOTS  is  not  included  in  the  FAR  definition  of  NDI.  GOTS  items  are  normally  produced  hv  rnm- 
mercial  production  services  that  would  be  obtained  through  commercial  item  FAR  procedures.  The 
policy  concerning  GOTS  production  implies  that  commercial  quality  procedures,  workmanship  prac¬ 
tices,  and  manufacturing  processes  would  be  used  in  contrast  to  extensive  acquirer-specified,  con¬ 
tractually  imposed  practices.  Government-developed  processes  should  be  disclosed  for  advice  only 
so  that  the  contractor  can  determine  the  best  means  of  providing  the  services  required,  with  the  suit¬ 
ability  of  the  end  item  the  sole  criterion  for  acceptance.  Where  a  GOTS  item  is  controlled  by 
detailed  design  engineering  packages  exclusively,  the  policy  requires  the  development  of  sufficient 
performance  specifications”  for  the  acquisition  and  acceptance  of  the  production  services. 

Although  this  may  seem  like  a  considerable  burden  for  the  acquisition  program,  the  information  is 
also  required  for  the  effective  life-cycle  support  of  the  item  and  must  be  available  in  any  case. 


WHY  NDI? 

Major  system  acquisitions  have  often  taken  a  dozen  years  or  more  to  design  and  field.  Even 
small  systems  costing  only  $10  million  to  develop  can  take  a  decade  to  move  from  the  designer’s 
desk  to  the  field.  Over  the  span  of  development  time,  the  requirements  of  those  in  the  field  are  not 
being  addressed.  Furthermore,  both  the  technologies  and  the  requirements  tend  to  change — often 
very  dramatically.  This  can  result  in  fielding  a  new  system  with  obsolescent  technology  that  does 
not  meet  current  threats  or  requirements.  The  ability  to  address  military  requirements  quickly  and 
effectively  directly  affects  national  security.  Even  when  the  newly  developed  system  does  meet  the 
threats  and  requirements,  an  extraordinary  expense  has  been  borne,  often  in  hidden  change  costs,  to 
keep  the  system  current.  Audits  of  newly  fielded  systems  have  revealed  that  as  much  as  60  percent 
of  the  development  costs  were  involved  in  making  changes  to  be  responsive  to  technology  and 
requirements  changes;  change  costs  driven  by  these  factors  in  the  30-percent  to  35-perce°nt  range  are 
very  common.  Clearly,  new  developments  always  cost  more  than  the  use  of  off-the-shelf  items.  As 
budgets  shrink  but  military  requirements  continue  growing  and  changing,  finding  solutions  more 
quickly  and  cost  effectively  becomes  imperative.  NDI  provides  a  means  of  achieving  high  levels  of 
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systems  effectiveness  quickly  and  cost  effectively.  Department  of  Defense  (DoD)  Directive  5000. 1 
recognizes  these  benefits  and  prioritizes  the  use  of  NDI  ahead  of  new  developments. 

Properly  applied  NDI  solutions  reduce  program  risks  through  the  increased  ability  to  be  responsive 
to  user  requirements,  but  risks  are  also  reduced  because  the  NDI  products  generally  have  a  field 
history  that  defines  their  realm  of  effective  application.  Developmental  systems  generally  require 
extensive  testing  to  develop  similar  levels  of  application  confidence.  In  changing  requirements  sce¬ 
narios,  open  architectures  substantially  reduce  the  costs  of  changes  needed  to  adapt  to  the  new  opera¬ 
tional  needs.  NDI-based  systems  generally  require  open  architectures  for  their  effective  integration; 
therefore,  adaptive  changes  in  NDI  systems  are  frequently  cheaper,  lower  risk,  and  quicker  to  imple¬ 
ment  than  new  developments. 

NDI  systems  enjoy  a  broad  application  base,  so  the  NDI  products  that  make  up  the  systems  are 
generally  large,  high-rate  productions  from  multiple  vendors.  Large,  high-rate,  competitive  produc¬ 
tion  bases  result  in  high  availability  of  products  and  product  support  items  at  low  cost.  Manufactur¬ 
ing  risk  and  the  risks  of  incorporating  technological  improvements  are  absorbed  by  the  vendors. 
Vendor  support  is  usually  very  responsive  since  the  market  share  often  depends  on  the  vendor’s  abil¬ 
ity  to  provide  responsive  support.  Product  quality  is  better  and  less  expensive  to  maintain  in  high- 
rate  production  lines.  Developmental  systems  almost  always  result  in  low-quantity  acquisitions  and 
very-low-quantity  reprocurements  for  support,  resulting  in  substantially  higher  support  costs  from 
single  sources  in  order  to  achieve  comparable  levels  of  quality  and  availability. 


HISTORICAL  BARRIERS  TO  USING  NDI 

With  these  benefits,  one  might  wonder  why  anybody  would  ever  develop  a  new  system  when  NDI 
is  available.  In  fact,  a  number  of  barriers  tend  to  divert  system  designers  away  from  NDI.  One  bar¬ 
rier  is  the  simple  fact  that  there  is  a  greater  benefit  to  the  design  agent  to  develop  the  product  rather 
than  use  NDI.  Product  design  allows  greater  degrees  of  design  control;  this  is  the  normal  excuse  put 
forward  for  not  using  NDI.  However,  a  potentially  large  profit  is  associated  with  product  design 
attached  to  the  cost-plus  environment  of  design  engineering,  and  the  risk  is  transferred  to  the  acquir¬ 
ing  agency  from  the  design  agent.  Also,  it  can  simply  be  more  technically  challenging  and  profes¬ 
sionally  fulfilling  to  design  the  product  than  to  use  somebody  else’s  solution. 

Another  barrier  is  the  set  of  unique  military  environments,  such  as  temperature  extremes,  very 
high  electromagnetic  interference  levels,  unusual  environmental  exposures,  the  effects  of  combat, 
and  unstable  power  systems.  When  first  analyzing  these  environmental  requirements,  it  is  easy  to 
assume  that  no  commercial  product  could  pass  the  imposed  rigors,  but  this  assumption  is  usually 
wrong.  Although  commercial  specifications  may  not  be  as  extreme  as  the  military  requirements,  the 
technologies  used  are  often  very  robust  and  can  easily  be  used  under  the  military  extremes  without 
modification  or  with  only  minor  modifications.  Of  course,  this  is  not  the  case  universally,  so  there  is 
extra  work  required  to  identify  potential  NDI  candidates  and  to  evaluate  the  potential  for  their 
application.  Extra  work  is  also  involved  to  resolve  the  differences  between  the  commercial  specifi¬ 
cations  used  to  develop  an  item  versus  the  military  specifications  associated  with  an  application. 

This  extra  work  needed  is  also  a  potential  barrier,  especially  if  there  is  a  predisposition  toward  build¬ 
ing  a  new  product  anyway.  After  all,  what  if  one  expends  this  extra  work  and  finds  out  that  no  suit¬ 
able  candidate  exists?  Then  resources  will  have  been  expended  without  a  tangible  result.  The  risk  of 
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pursuing  an  NDI  alternative  is  usually  very  small  but  also  very  real;  system  designers  often  overesti¬ 
mate  this  risk  (often  unintentionally)  in  order  to  justify  new  developments. 

Strangely,  one  of  the  main  strengths  of  NDI — quick  reaction  to  satisfying  requirements — can  also 
be  a  barrier.  It  is  much  more  difficult  for  an  organization  to  perform  a  series  of  quick-reaction,  rela¬ 
tively  low-dollar  programs,  than  to  work  a  single  long-term  program.  Even  with  political  problems 
of  maintaining  a  funding  line  for  10  to  12  years,  it  is  much  easier  and  more  secure  to  do  the  long¬ 
term  program  than  to  obtain  funding,  plan,  and  accomplish  5  to  8  short-term  programs  over  the  same 
length  of  time.  This  problem  is  mitigated  where  the  acquisition  community  is  supported  by  a  contin¬ 
uing  line  item  of  budget;  however,  this  is  not  the  case  for  the  majority  of  system  needs. 

Acquisition  community  managers  may  also  be  put  off  by  the  fact  that  many  of  the  life-cycle  costs 
associated  with  NDI  are  not  markedly  different  from  developed  systems,  although  the  cost  distribu¬ 
tion  may  be  quite  different.  This  can  occur  because  NDI  product  life  spans  may  be  only  3  to  6  years 
and  require  frequent  system  upgrades.  Normal  engineering  changes  to  a  developed  system  will 
result  in  a  10-percent  change  per  year  in  the  product  baseline;  NDI  systems  may  see  a  16-  to 
30-percent  change  per  year.  (On  the  other  hand,  system  architectures  designed  to  accommodate 
NDI  tend  to  lend  themselves  to  easy  technological  upgrades  very  inexpensively.  Systems  consisting 
primarily  of  developed  products  also  need  to  be  upgraded  with  new  technologies,  and  these  upgrades 
may  be  significantly  more  expensive  to  maintain  per  unit  of  functionality.)  Acquisition  managers 
need  to  take  the  full  life-cycle  cost/total  cost  of  ownership  impact  into  account  when  deciding^for  or 
against  a  NDI  acquisition  strategy.  The  initial  procurement  cost  savings  can  only  be  preserved  in 

NDI  acquisitions  when  the  system  upgrade  factors  are  appropriately  accounted  for  in  the  acquisition 
planning. 

Furthermore,  the  military  support  systems  are  “tuned”  to  support  developed  products  rather  than 
NDI.  NDI  systems  can  be  in  the  field  for  years  and  even  be  approaching  the  end  of  product  life  prior 
to  reaching  a  designated  support  date.  While  shorter  support  dates  may  be  available  using  COTS/ 
NDI  support  procedures,  these  procedures  are  still  in  development.  This  fact  interposes  significant 
challenges  to  system  designers  to  plan  for  logistics  support,  and  many  designers  do  not  want  to  be 
bothered  with  the  added  headaches.  Consideration  of  NDI  is  often  viewed  as  just  additional  work 

that  is  not  contributing  to  “making  the  dirt  fly,”  so  the  potential  benefits  are  sometimes  ignored  rather 
than  pursued. 

So,  why  NDI? — faster,  more  effective  solutions  at  lower  cost  and  risk.  However,  NDI  solutions 
are  not  always  available  or  appropriate.  Nevertheless,  NDI  should  always  be  considered  in  system 
architectures  and  system  designs  during  the  requirements  definition  and  conceptual  phases  of  an 
acquisition. 
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COMMERCIAL  OFF-THE-SHELF  (COTS) 


COTS  STANDARDS  DEFINED— DIFFERENT  GRADES  OF  COTS 

The  designation  of  COTS  is  frequently  misunderstood.  First  of  all,  the  term  “commercial”  is  often 
thought  of  as  merely  anything  produced  by  industry  (or  a  nongovernmental  entity).  Actually,  “com¬ 
mercial”  refers  to  a  product  generated  in  conformance  to  commercial  specifications.  There  are  three 
different  kinds  of  commercial  specifications:  (1)  company  proprietary  specifications,  (2)  market 
standards,  and  (3)  industry  standards.  Each  of  these  forms  of  commercial  specification  has  its  own 
unique  properties  that  influence  a  product’s  potential  viability  for  military  applications.  Secondly, 
“off-the-shelf’  implies  that  the  product  is  in  production.  As  noted  in  the  definitions  of  NDI  above,  a 
distinction  is  made  between  COTS  and  NewCOTS  with  regard  to  the  product  maturity.  These  fac¬ 
tors,  the  form  of  commercial  specification  and  the  product  maturity,  combine  to  influence  product 
quality,  product  life,  depth  and  availability  of  documentation,  reliability,  supportability,  and  training 
suitability.  In  addition,  the  COTS  source  of  supply  may  be  either  foreign  or  domestic.  All  of  these 
elements  are  important  to  take  into  account  when  assessing  products  for  use  in  a  system  application. 

Many  of  the  military  specifications  were  initially  generated  to  overcome  the  problems  associated 
with  commercial  specifications.  Most  of  these  problems  are  associated  with  company  proprietary 
specifications  or  with  market  standards.  Most  of  the  problems  are  generated  by  highly  variable  qual¬ 
ity  and  reliability  factors  that  drive  life-cycle  support  decisions.  Military  specifications  fixed  these 
factors  to  enable  support  planners  to  complete  their  tasks.  Commercial  specifications  can  lead  to 
very  great  variances  in  these  areas.  Quality  and  reliability  factors  are  often  hard  to  accurately  esti¬ 
mate  for  support  planning  purposes  for  commercially  specified  products. 

Market  standards  are  the  most  difficult  to  quantify  because  the  design  practices  are  driven  by  cost 
factors  to  produce  the  most  acceptable  product  for  the  market  place  at  the  lowest  possible  price. 
Market  acceptability  may  demand  very  high  quality,  as  in  medical  life  support  equipment,  or  may 
allow  very  low  quality  in  order  to  be  inexpensive,  as  in  expendable  electronic  watches  or  calculators. 
To  assess  products  designed  for  market  standards,  it  is  necessary  to  understand  the  market  forces. 
This  may  be  relatively  easy  for  long-term,  well-established  markets  that  are  evolving  slowly,  but  it 
may  be  very  difficult  for  new  or  highly  dynamic  markets.  In  addition,  companies  competing  in  a 
market  may  have  different  market  perspectives,  leading  to  radically  different  design  approaches  and 
pricing  structures.  It  is  necessary  to  adequately  define  the  commercial  market  for  any  COTS  prod¬ 
uct.  Otherwise,  the  initial  assessment  of  the  viability  of  COTS  products  will  be  faulty.  For  instance, 
it  would  not  be  adequate  to  do  a  market  assessment  of  the  personal  computer  market  as  a  whole;  it 
would  generally  be  necessary  to  distinguish  between  segments  of  the  market  and  to  identify  those 
segments  most  closely  approximating  the  target  system  application.  Good  market  identification 
allows  sufficient  analysis  to  be  done  to  determine  the  quality  and  cost  factors  behind  the  market  seg¬ 
ment.  Having  identified  the  quality  and  cost  factors,  it  is  then  possible  to  project  the  likely  variables 
in  specification  factors  (including  those  that  affect  supportability)  that  are  critical  to  a  particular 
application. 

A  major  part  of  the  variability  in  market  standards  is  termed  “best  commercial  practices”  work¬ 
manship.  Most  proprietary  specifications  have  well-defined  company  practices,  and  industry  stan¬ 
dards  usually  have  quality  verified  by  test  specifications.  Market  specifications  and  the  associated 
commercial  practices  are  driven  by  market  cost.  The  best  commercial  practice  for  a  specific  market 
is  defined  by  what  is  acceptable  quality  and  by  lowest  possible  costs.  In  markets  that  involve  auto¬ 
mated  high-volume  productions,  the  quality  and  workmanship  is  often  defined  through  the 
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production  processes.  However,  most  products  are  not  produced  by  extremely  automated  lines,  and 
high  volumes  are  often  achieved  through  the  use  of  cheap  offshore  labor.  This  can  lead  to  very  high 
variability  in  workmanship  standards  and  high  risks  associated  with  support  decisions  driven  by 
quality  factors.  Indeed,  the  best  commercial  practice  for  some  market  segments  may  be  very  low 
quality  while  a  different  related  market  segment  might  enjoy  a  very  high  quality,  simply  because  the 

quality  versus  cost  decisions  are  so  different  for  the  customer  bases  that  define  those  market  set^- 
ments.  “ 


Proprietary  specifications  may  have  excellent  performance,  quality,  and  reliability  characteristics 
but  still  have  poor  supportability.  Most  proprietary  specifications  are  closely  held  by  the  source  of 
supply  and  are  difficult  to  document  for  purposes  of  training  and  maintenance.  This  results  in  very 
high  levels  of  assembly  for  repair  and  provisioning  purposes  (i.e.,  highly  complex  and  expensive 
lowest  replaceable  units).  The  resulting  life-cycle  costs  may  be  high  because  of  the  number  of 
expensive  pipeline  spares  required,  or  the  system  availability  may  be  very  low  due  to  downtime 
awaiting  parts.  This  is  even  more  difficult  when  a  foreign  source  of  supply  is  involved.  Products 
having  proprietary  specifications  are  usually  only  available  from  a  single  source  of  supply.  How¬ 
ever,  there  are  a  few  exceptions  where  an  “inventor”  source  of  supply  has  licensed  other  manufactur¬ 
ers.  Several  unique  (and  mutually  incompatible)  product  designs  may  be  competing  in  the  same 
market.  Here,  the  system  designer  must  analyze  the  market  and  the  competing  company  marketing 
strategies  as  well  as  the  technology  embodied  in  the  product  to  determine  product  viability  over  the 
long  term.  A  classic  example  is  the  VCR  market  in  1980  where  several  formats  competed,  eventu¬ 
ally  won  by  the  VHS  format  in  spite  of  superior  quality  and  performance  elements  in  the  Beta  for¬ 
mat.  A  different,  but  closely  related  market,  is  the  camcorder  market  now  dominated  by  the  8-mm 
format,  with  VHS  and  VHS-C  formats  holding  a  market  niche.  Good  technical  logic  does  not  define 
these  markets;  but  price,  product  availability,  consumer  support,  convenience,  and  other  factors  play 
key  roles.  Most  of  these  factors  do  not  relate  to  military  applications.  Nevertheless,  the  products  in 
these  markets  do  have  utility  in  some  military  applications. 

Industry  standards  are  specifications  subscribed  to  and  supported  by  a  large  number  of  manufac¬ 
turers  responding  to  a  published  industry  standard.  Many  industry  standards  have  been  adopted  by 
DoD  because  the  same  (or  higher)  quality  and  performance  factors  can  be  achieved  as  for  military 
specifications  but  at  a  much  lower  cost.  In  addition,  many  industry  standards  are  derived  from  gov¬ 
ernment-initiated  work  and  become  a  marketplace  consensus  standard  that  is  only  partially  docu¬ 
mented  in  a  formal  specification  forum.  Companies  may  conform  to  the  standard  but  add  special 
features  or  “flavors”  to  give  their  products  a  market  edge.  Some  interface  standards  such  as  the  vari¬ 
ous  Internet  protocols  could  be  considered  to  fall  into  this  category.  Variations  in  industry  standards 
are  also  illustrated  by  the  various  UNIX  flavors  and  variations  on  instrumentation  control  bus  stan¬ 
dards  (IEEE-488  versus  HP-IB  versus  TEK-488,  etc.).  Unfortunately,  some  of  these  special  fea¬ 
tures  become  essential  to  the  applications  using  the  standards,  so  the  application  using  one  enhance¬ 
ment  is  no  longer  interoperable  with  an  application  using  a  different  flavor  of  the  same  standard. 

Also,  some  of  the  enhanced  features  are  difficult  to  differentiate  from  the  true  standard  features,  so 
application  designers  have  a  very  difficult  time  limiting  their  implementation  to  a  true  standard 
implementation  that  will  be  interoperable. 

The  conamercial  standards  issue  is  even  more  complex  when  foreign  sources  of  supply  are  consid¬ 
ered.  Many  FOOTS  designs  are  coordinated  proprietary  standards  because  the  foreign  government 
or  a  government-coordinated  industry  group  has  agreed  to  the  standards,  but  the  design  disclosure  is 
still  company  proprietary.  Usually,  an  intergovernmental  agreement  is  required  to  license  a  domestic 
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source  of  supply  or  even  a  domestic  source  of  maintenance  and  training.  Such  agreements  are  not 
always  feasible.  FCOTS  built  to  the  various  ISO  standards  are  preferred. 

It  is  important  to  characterize  the  commercial  market  segment  for  which  a  product  is  provided,  to 
document  differences  between  the  commercial  marketplace  and  the  program  acquisition  require¬ 
ments,  and  to  recognize  the  standards  employed  in  the  product  design  and  production.  This  informa¬ 
tion  is  needed  to  determine  the  appropriate  approaches  to  the  acquisition  of  the  functional  item  and 
for  its  support.  For  instance,  an  item  having  low  quality  (high  variability  in  performance)  may  be 
acceptable  if  the  parent  system  requirements  can  be  designed  to  accept  the  range  of  variability,  but  it 
is  more  commonly  necessary  to  provide  a  screening  acceptance  test  to  be  applied  by  either  the  source 
of  supply  or  by  the  receiving  agency  (together  with  a  suitable  warranty  agreement  to  cover  discrep¬ 
ant  items).  Products  from  one  market  niche  may  be  driven  by  different  factors  than  those  from 
another  segment,  so  key  elements,  such  as  the  availability  of  information  needed  for  support  plan¬ 
ning  or  the  different  flavors  of  industry  standards  or  the  range  of  contractor  support  services,  may 
depend  on  market  elements  that  are  not  common  between  the  sources  of  supply.  This  will  drive  how 
procurement  documentation  is  assembled,  how  support  plans  are  generated,  and  how  the  source 
selection  is  conducted. 

COTS  MATURITY 

Product  maturity  is  very  important  in  making  system  support  decisions  in  the  use  of  COTS.  Prod¬ 
uct  maturity  is  a  function  of  production  volume  and  years  of  service.  Products  that  have  been  in  pro¬ 
duction  and  use  for  many  years  have  an  experience  base  that  allows  support  decisions  to  be  made 
with  relatively  low  risk.  In  addition,  the  design  requirements  embodied  in  the  product  design  have 
evolved  in  a  mature  design,  so  the  levels  of  quality,  reliability,  and  performance  tend  to  be  both  very 
stable  and  relatively  high.  Documentation  for  interfacing  to  mature  products  tends  to  be  readily 
available.  Commercial  support  tends  to  be  available  and  affordable.  Even  for  high-technology  prod¬ 
ucts  that  are  susceptible  to  frequent  product  improvements,  the  cost  of  maintaining  a  system  through 
repair  and  selective  upgrades  is  very  affordable. 

On  the  other  hand,  many  mature  products  may  lack  the  high-technology  performance  edge  that 
may  be  required  in  military  applications.  Mature  products  may  also  be  based  on  industry  standards 
that  are  being  superseded  by  newer  technology  standards,  so  the  remaining  product  life  may  be  rela¬ 
tively  short.  As  the  transition  in  standards  takes  place,  companies  pull  out  of  the  market,  resulting  in 
fewer  alternate  sources,  reduced  availability  of  support,  reduced  operational  availability,  and  high 
risks  of  obsolescence.  The  transition  to  replacement  technology  standards  often  introduces  higher 
life-cycle  costs  as  a  system  is  upgraded  than  for  COTS  products  not  near  their  end-of-market  life. 

These  kinds  of  obsolescence  problems  create  pressure  to  use  NewCOTS  products  that  lack  matu¬ 
rity.  Products  that  are  immature  because  of  the  lack  of  production  volume  tend  to  have  higher  vari¬ 
ability  in  quality,  leading  to  uncertainties  and  inefficiencies  in  planning  support.  Also,  most  imma¬ 
ture  products  have  insufficient  documentation,  test  data,  and  field  data  from  which  to  derive  training 
and  support  information.  This  does  not  mean  that  NewCOTS  products  should  not  be  considered,  but 
it  does  mean  that  added  work  is  required  by  the  system  acquisition  agent  to  compensate  for  these 
characteristics.  This  additional  work  may  be  in  the  form  of  added  market  research,  tailored  screen 
tests,  reallocation  of  system  requirements  to  compensate  for  product  variability,  added  procurement 
requirements  to  reduce  the  consumer  risks,  developing  contingency  support  plans,  or  some  combina¬ 
tion  of  all  of  these  techniques. 
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Companies  will  often  advertise  a  product  prior  to  its  actual  design  and  production  simply  to  find 
out  if  a  market  exists.  If  they  get  sufficient  inquiries  or  even  orders,  then  they  will  invest  in  product 
development.  This  often  means  that  the  first  production  units  are  really  hand-built  prototypes.  Sub¬ 
sequent  manufacture  of  the  item  may  result  in  design  changes  that  make  the  original  units  obsolete 
and  unsupportable  (spare  parts  may  be  incompatible;  documentation  frequently  is  not  accurate). 
Often  the  original  customers  are  also  the  “field  test  agency”  and  may  suffer  many  product  introduc¬ 
tion  pains.  NewCOTS  products  may  also  be  changed  in  order  to  take  advantage  of  new  or  evolving 
industry  standards,  causing  the  original  designs  to  become  obsolete.  New  production  processes  may 
be  brought  to  bear  that  may  change  product  tolerances  or  introduce  second-order  effects  that  must  be 
incorporated  into  system  documentation.  Both  NewCOTS  and  mature  COTS  will  be  continually 
incorporating  design  changes,  but  these  changes  will  usually  be  forced  by  different  market,  eco¬ 
nomic,  and  technical  issues.  The  primary  difference  between  NewCOTS  and  mature  COTS  is  that 
the  design  changes  incorporated  in  the  normal  course  of  the  product  evolution  tend  to  cause  ma  jor 
supportability  problems  for  NewCOTS,  but  not  for  more  mamre  designs. 

Some  technologies  evolve  so  rapidly  that  the  product  life  is  limited  to  as  little  as  6  months.  In 
these  technologies,  the  next  generation  is  in  development  before  the  current  generation  is  even  in 
high-volume  production.  Many  risks  are  introduced  through  products  incorporating  high-technology 
elements.  These  risks  can  only  be  mitigated  through  careful  system  life-cycle  planning  and  manage¬ 
ment  that  continuously  monitors  the  technology  market.  System  planners  can  also  reduce  risk  by 
actively  sharing  field  experience  and  requirements  information  with  companies  developing  new 
products  in  the  market.  Although  the  system  management  costs  are  potentially  significant  to  the  pro¬ 
gram,  the  life-cycle  cost  savings  are  substantial  because  pathways  exploiting  the  rapidly  changing 
market  standards  can  be  documented  to  avoid  significant  future  costs. 

COTS  products  have  a  range  nearly  as  extensive  as  NDI  products.  It  is  necessary  to  understand 
the  characteristics  of  a  specific  COTS  product  and  its  associated  market  in  order  to  accurately  assess 
the  product  utility  for  a  particular  application  and  to  support  the  item  in  the  field.  There  is  a  continu¬ 
ing  need  to  survey  the  market  from  the  origination  of  a  product  through  the  entire  system  life-cycle 
support  phase. 


A  PROCESS  TO  BUY  NDI 

The  process  described  herein  is  one  of  many  possible  processes  for  buying  NDI;  however,  it  is 
crafted  to  be  both  natural  and  compatible  with  good  processes  for  developing  systems.  This  results 
in  an  approach  that  results  in  an  NDI-based  system  whenever  good  alternatives  are  available  and  an 
open-architecture  system  development  whenever  NDI  alternatives  are  not  available.  A  closed  archi¬ 
tecture  development  will  only  result  when  open  standards  are  not  available  or  when  a  conscious  deci¬ 
sion  is  made  toward  a  closed  architecture.  Further  information  on  this  process  is  provided  in  Techni¬ 
cal  Document  108.  The  process  flows  as  follows: 

1.  Define  and  analyze  mission  requirements. 

2.  Determine  system  functional  characteristics  (and  alternatives  thereof). 

3.  Define  one  or  more  technical  approaches,  giving  preferences  to  open  architecture  approaches. 

4.  Define  top-level  system  partitions  of  functionality  along  natural  lines  of  requirements  (such  as 
operational  differences  and  environmental  differences). 
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5.  Partition  the  system  through  successively  lower  levels  of  functionality,  supporting  decisions 
with  interface  standards  selections  prioritized  by  open  and  industrial  standards.  Consider 
technology  factors  and  perform  a  market  analysis  of  potential  NDI  at  each  level. 

6.  Document  the  resulting  system  specifications  and  obtain  industry  comments  where 
appropriate. 

7.  Conduct  screens  of  potential  NDI  products,  documenting  integrated  logistics  impacts  and  sup¬ 
port  decision  constraints.  [Note;  A  tailored  screen  may  be  conducted  either  independently  or 
in  conjunction  with  the  acquisition — see  TAILORED  SCREENS  FOR  NDI.] 

8.  Complete  the  procurement  package  and  proceed  with  the  acquisition,  including  support.  [Note; 
When  NDI  products  are  not  available,  the  procurement  package  will  include  development 
specifications.] 

DETERMINING  REQUIREMENTS 

It  is  always  important  to  accurately  and  completely  analyze  and  characterize  the  requirements  of 
the  system;  this  is  even  more  essential  for  NDI  acquisitions.  A  failure  to  establish  requirements  will 
lead  to  system  designs  that  exclude  NDI  solutions  or  that  incorporate  NDI  of  inferior  quality  for  the 
application.  In  setting  the  system  requirements,  it  is  important  to  characterize  the  mission  require¬ 
ments  and  top-level  technical  requirements  in  ways  that  do  not  dictate  a  military  proprietary  design. 
The  system  specifier  needs  to  state  the  mission  and  operational  requirements  in  a  way  that  can  be 
accurately  interpreted  by  product  suppliers.  This  usually  involves  transforming  operational  require¬ 
ments  into  their  direct  technical  equivalents  while  avoiding  the  trap  of  framing  those  requirements  in 
the  terms  of  a  specific  system  architecture.  Top-level  technical  requirements  do  not  dictate  one 
architecture  over  another,  but  allow  the  flexibility  for  the  system  designer  to  choose  one  or  more  sys¬ 
tem  architectures  as  a  function  of  the  system  partitioning.  (See  SYSTEM  PARTITIONING  FOR 
NDI). 

Ideal  requirements  are  well  defined  and  stable.  It  is  often  not  possible  to  define  and  stabilize  all  of 
the  requirements;  nevertheless,  it  is  almost  always  possible  to  bound  areas  of  requirements  that  are 
vague  or  variable.  Merely  bounding  the  requirements  areas  often  allows  system  design  efforts  to 
proceed  with  firm  technical  requirements.  Different  classes  of  requirements  must  be  defined  during 
the  requirements  definition  phase  or  early  in  the  conceptual  phase  of  a  system  acquisition  prior  to 
actually  determining  the  system  architecture  or  doing  the  system  partitioning.  Although  there  are 
numerous  ways  of  defining  these  classes,  the  following  list  has  been  found  to  be  practical; 

Performance  Requirements 

Combat  capabilities 

Survivability  capabilities 

Interoperability  requirements 

Safety  requirements  (includes  personnel,  product,  and  environmental  system  safety 
criteria) 

Security  requirements  (physical,  operational,  electronic,  cryptological,  etc.) 

Other  functional  requirements 
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Usage  Requirements 

Usage  mode  (fixed,  airborne,  shipbome,  deployable,  portable,  etc.) 

Usage  constraints  (“must  have”  versus  “nice  to  have”  features) 

Frequency  band  operating  requirements,  including  spectrum  operating  rules 

Security  code  usage  requirements 

Satellite  resource  allocation  requirements 

Operating  duty  cycle  requirements 

Mission  profile  requirements 

Concept  of  operation  requirements/business  practice  requirements 
Concept  of  employment/concept  of  operation  constraints 

Environmental  Requirements 

Operating  conditions 
Non-operating  conditions 

Combat  conditions  (conventional,  nuclear,  biological/chemical) 

Storage  and  shipping  conditions 
Maintenance  and  repair  conditions 

Installation  Requirements 

Space  and  location  requirements 
Weight  and  moments  requirements 
Power  system  interfaces 

Other  support  system  interfaces  (such  as  dry  air,  precise  time,  stable  element  outputs, 
etc.) 

Human  Factors  Requirements 

Operator  personnel — fully  trained 

Operator  personnel — reduced  grade  and  training  levels 

Operator  personnel — combat  stressed  or  fatigued 

Organizational  maintenance  personnel — fully  trained 

Organizational  maintenance  personnel — ^reduced  grade  and  training  levels 

Organizational  maintenance  personnel — combat  stressed  or  fatigued 

Intermediate  maintenance  personnel 

Depot  maintenance  personnel 

Installation  personnel 

Electromagnetic  Compatibility  Requirements 

Electromagnetic  interference — susceptibility  (conducted  and  radiated) 

Electromagnetic  interference — conformance  (conducted  and  radiated) 

Electromagnetic  pulse 
TEMPEST  criteria 

Hazardous  electromagnetic  radiation  (fuels  (HERF),  ordnance  (HERO),  and  personnel 
(HERP)) 

Electrostatic  discharge  susceptibility/protection 

Shielding,  bonding,  and  grounding  requirements 

Abnormal  conditions  (lightning,  power  system  spikes  and  transients) 
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Supportability  Requirements 

System  life  requirements 

Operational  availability  requirements 

Mission  reliability  requirements 

Life-cycle  (support)  cost  constraints/requirements 

Fault/failure  criteria 

Maintenance  criteria 

Downtime  constraints/requirements 

Maintainability  characteristics  and  requirements 
Downtime  for  parts  requirements 
Downtime  for  assistance  requirements 
Administrative  downtime  requirements 
Personnel  constraints/requirements 
Training  constraints/requirements 
Packaging,  packing,  and  preservation  requirements 
Handling,  storage,  and  transportation  constraints  and  requirements 
Limitations  on  evacuation  of  repairables 
Interchangeability  of  spares  requirements/constraints 
Operational  logistics  constraints 
Technical  data  constraints/requirements 

Production  Requirements 

Initial  quantities  and  rates 
Follow-on  quantities  and  rates 

Foreign  military  sales/cooperative  production  agreement  requirements 
Spares  acquisition  requirements 

Technical  Interface  Requirements 

Interface  standards  and  protocols 
Computer  operating  system  compatibility 

Minimum  computer  hardware  support  standards  (speed,  processor  type,  output  ports, 
etc.) 

Minimum  expansion  capabilities 
Programmatic  Requirements 

Affordability  criteria  (both  acquisition  and  life-cycle  support) 

Crown  Jewel  performance  criteria  (priority  user  requirements  and  minimum  acceptable 
criteria) 

Schedule  constraints  (such  as  delivery  prior  to  a  specific  mission  or  operation) 

Risk  constraints 

Each  class  of  requirements  contains  elements  critical  to  the  overall  effectiveness  and  suitability  of  the 
system.  These  requirements  must  be  identified  for  system  test  planning  as  well  as  for  system  design 
and  acquisition.  The  requirements  apply  whether  the  system  is  to  be  designed  from  the  ground  up, 
integrated  from  existing  components,  adapted  from  off-the-shelf  items,  or  some  hybrid  of  these 
approaches.  If  the  requirements  are  properly  defined,  none  of  these  approaches  will  be  precluded. 
Also,  these  requirements  are  interactive  with  each  other,  so  the  requirements  relationships  must  be 
determined  at  least  qualitatively,  although  quantitative  relationships  are  desired  and  often  must  be 
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derived  in  order  to  complete  a  system  design.  Doing  a  thorough  job  of  requirements  definition 
greatly  contributes  to  making  build/buy/modify  decisions  during  the  system  partitioning  phases  of 
system  design. 

Military  applications  do  have  extremes  in  performance  envelopes,  environmental  requirements, 
and  support  requirements  that  can  greatly  exceed  normal  commercial  specifications.  Nevertheless, 
these  extremes  are  usually  not  experienced  simultaneously  nor  for  long  periods  of  time,  so  the 
product  stresses  are  usually  within  the  technical  capabilities  of  high-quality  products  built  to  com¬ 
mercial  specifications.  Also,  environmental  extremes  in  military  platforms  are  usually  isolated  to 
relatively  small  areas  of  the  platform,  and  combat  extremes  are  usually  short  in  duration.  The  envi¬ 
ronmental  requirements  for  military  applications  are  often  used  to  exclude  COTS  products  from  con¬ 
sideration  even  though  the  actual  environment  on  the  platform  may  be  very  similar  or  even  less  stres¬ 
sing  than  some  commercial  environments.  For  example,  the  simultaneous  extremes  of  high 
temperature  and  high  humidity.  Navy -peculiar  requirements,  rarely  occur  aboard  Navy  ships  and  are 
confined  to  limited  areas  of  the  engine  rooms  when  they  do  occur.  Most  combat  spaces  must  be  con¬ 
trolled  to  less  stressful  conditions  than  commercial  specifications  because  people  must  be  combat 
effective  in  these  spaces.  Although  mission  profile  analysis  may  provide  useful  insights  to  combined 
environmental  stresses,  field  measurements  are  usually  required  to  determine  the  specific  environ¬ 
mental  limits. 


The  most  common  ureas  left  underdefined  are  in  the  areas  of  human  factors  requirements  and  sup- 
portability  requirements.  The  failure  to  adequately  define  these  requirements  leads  to  high  life-cycle 
costs  no  matter  what  kind  of  system  is  being  acquired.  Inadequate  information  will  be  available  to 
the  system  designers  and  the  subsequent  design  engineers  developing  a  product  or  evaluating  NDI 
alternatives,  resulting  in  poor  decisions  in  the  system  life-cycle  support  area.  Most  commonly,  sup¬ 
port  requirements  u  ill  dictate  higher  levels  of  operational  availability  than  will  result  from  normal 
support  decisions.  Tlie  use  of  COTS  can  be  very  expensive  when  it  is  necessary  to  achieve  high 
operational  availahil:t>  performance.  When  costs  of  support  are  controlled,  operational  availability 
will  suffer  because  ol  eseessive  downtimes.  NonCOTS  NDI  forms  may  also  be  similarly  affected. 
This  is  also  a  failing  oi  most  newly  designed/nonNDI  systems.  The  failure  to  adequately  define 
these  requirements  often  results  from  acquisition  managers  failing  to  recognize  the  importance  of  the 
requirements,  thereby  failing  to  task  and  to  fund  the  efforts  needed  to  do  the  requirements  analyses  in 
these  areas.  Also,  these  requirements  areas  can  be  rather  vague  and  can  lack  well-recognized  quanti¬ 
tative  requirement  statements,  resulting  in  even  more  effort  to  properly  bound  the  requirement  vari¬ 
abilities  and  to  define  the  requirements  in  usable  quantitative  terms. 

In  even  the  most  state-of-the-art  systems,  at  least  80  percent  of  the  system  functionality  has 
already  been  done  before.  If  the  functionality  has  been  done  before,  it  almost  always  exists  in  a  NDI 
form;  therefore.  NDI  candidates  ought  to  be  considered  as  a  routine  part  of  a  system  design  imple¬ 
menting  virtually  an\  .set  of  requirements.  Only  requirements  that  are  truly  new  or  that  need  to  be 
advanced  by  the  state  of  the  art  should  require  new  development. 

While  substantial  portions  of  the  system  functionality  have  already  been  done  before,  it  is  impor¬ 
tant  to  analyze  the  operational  requirements  in  a  way  that  does  not  automatically  assume  that  per¬ 
forming  operations  “the  same  old  way”  is  the  correct  approach.  Operational  requirements  analysis 
should  start  with  the  mission  definition  in  its  broadest  terms.  Failing  to  take  this  approach  will  result 
in  “same  old  way”  operational  requirements  that  will  preclude  the  best  use  of  available  technology. 
Some  of  the  best  system  designs  anticipate  how  technology  will  enable  future  changes  in  concepts  of 
operations  and  usage  doctrine.  This  implies  a  large  amount  of  concurrency  in  the  analysis  of  the 
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operational  requirements,  the  top-level  system  partitioning  decisions  (see  SYSTEM  PARTITION¬ 
ING  FOR  NDI),  and  the  assessment  of  technology  implicit  in  conducting  a  market  survey  as  a  part 
of  performing  the  system  partitioning. 

In  defining,  analyzing,  and  determining  requirements  of  all  forms,  it  is  also  useful  to  determine  the 
various  methods  that  might  be  employed  to  verify  the  requirements.  This  analysis  can  be  used  to 
determine  the  most  effective  means  of  achieving  the  desired  system  quality.  The  avenues  of 
approach  may  vary  from  acceptance  of  the  market  quality  to  extensive  testing  tailored  to  qualify  the 
products  for  the  application.  Effort  in  the  area  requirements  analysis  and  verification  early  in  a  proj¬ 
ect  will  save  orders  of  magnitude  in  future  costs  over  a  system  life  cycle. 


SYSTEM  PARTITIONING  FOR  NDI 

System  partitioning  is  the  art  of  allocating  system  functionality  (and  other  requirements)  into  a 
hierarchy  of  product  cells.  Top-level  cells  are  the  most  complex  and  contain  the  most  functionality, 
while  the  subsidiary  cells  are  relatively  non-complex  and  contain  limited  functional  implementations. 
The  traditional  levels  of  complexity  are  system,  subsystem,  set,  group,  unit,  assembly,  subassembly, 
module,  and  piece  part.  The  allocation  of  requirements  to  these  levels  also  defines  levels  of  repair, 
levels  of  standardization,  levels  of  design  ownership,  and  the  levels  at  which  the  acquirer  (govern¬ 
ment)  is  responsible  for  system  integration  decisions  versus  the  supplier.  Each  act  of  partitioning 
creates  a  series  of  internal  system  interfaces  and  defines  a  piece  of  the  system  architecture.  Gener¬ 
ally,  the  cells  interface  with  each  other  at  the  same  level,  so  partitioning  decisions  can  be  made  inde¬ 
pendently  for  one  part  of  a  system  than  for  another  part.  Even  when  one  portion  of  a  system  is  not 
amenable  to  NDI,  that  does  not  preclude  the  use  of  NDI  in  other  portions  of  that  system. 

The  partitioning  decisions  consider  the  requirements  flowed  down  to  the  level  of  complexity  at 
which  the  decision  is  being  made,  plus  an  assessment  of  available  technologies  needed  to  implement 
those  requirements.  The  assessment  of  available  technologies  takes  into  account  any  products  that 
address  similar  functional  requirements.  The  step  often  missing  in  this  assessment  process  is  the  one 
that  analyzes  each  product  for  its  viability  and  utility  in  addressing  the  full  set  of  requirements  at  that 
level.  The  conduct  of  this  assessment  is  greatly  enhanced  by  good  requirements  flowdown  processes 
and  tools.  A  good  knowledge  of  the  underlying  market  associated  with  the  product  is  also  needed. 
The  technology  assessment  often  does  not  include  NDI  products  because  the  market  knowledge  is 
not  immediately  available.  Good  partitioning  processes  also  look  ahead  to  the  standards  and  prod¬ 
ucts  that  exist  at  lower  levels  of  complexity  that  are  exposed  when  one  decision  is  made  versus 
another.  The  look-ahead  process  helps  to  avoid  deciding  to  favor  a  good  proprietary  design  that 
implements  a  very  high  level  of  standardization  when  excellent  and  more  cost-effective  designs 
could  be  easily  implemented  at  a  lower  level  of  complexity.  NDI  can  also  be  neglected  when  good 
requirements  flowdown  processes  are  not  being  used,  hampering  the  ability  to  look  ahead  to  the 
impact  of  various  standards  at  lower  levels  of  complexity. 

The  system  partitioning  decisions  should  be  looking  ahead  to  NDI  use,  consistent  with  the  DoD 
policy  that  establishes  an  order  of  preference  for  NDI  as  follows: 

•  (Non-materiel  options  not  requiring  item  acquisitions.) 

•  MILOTS,  not  involving  additional  production. 

•  Modified  MILOTS,  not  involving  additional  production. 
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•  Other  _OTS,  especially  commercial  items  (as  defined  by  the  FAR). 

•  NDI  (unmodified,  including  MILOTS)  (as  defined  by  the  FAR). 

•  FME. 

•  Modified  _OTS  or  FME. 

•  Integrated  NDI. 

•  (Joint-Service  Development). 

•  (Service-unique  Development). 

This  order  of  preference  should  be  considered  at  each  iteration  of  the  system  partitioning  process. 
Costs,  risks,  supportability,  safety,  and  human  factors  should  be  considered  in  evaluating  each  alter¬ 
native.  Large  or  complex  systems  may  be  an  integration  of  several  of  the  above  categories. 

SYSTEM  ARCHITECTURES 

A  system  architecture  summarizes  the  character  of  the  standards  that  interface  the  cells  of  the  sys¬ 
tem  partitions.  The  system  architecture  is  actually  defined  by  the  common  philosophy  used  in  mak¬ 
ing  the  system  partitioning  decision  rather  than  being  an  imposed  structure.  When  a  system  design  is 
initiated,  there  will  be  architectural  policies;  however,  the  actual  architecture  will  be  consistent  with 
those  policies  only  to  the  degree  that  the  system  designers  use  standards  having  characteristics  that 
agree  with  the  policies.  In  fact,  this  might  not  even  be  possible  in  those  cases  where  the  technology 
is  not  supported  by  standards  consistent  with  the  policies. 

There  are  two  basic  types  of  system  architectures:  open  and  closed.  An  open  system  architecture 
consists  of  standards  that  are  available  for  public  use.  Availability  for  public  use  means  that  the 
standard  information  is  published  in  a  public  forum,  that  the  standard  can  be  used  without  legal 
restrictions,  that  the  interfaces  are  fully  characterized,  and  that  the  technology  is  publicly  accessible. 
All  other  architectures  are  closed  because  the  interfacing  standards  must  be  obtained  through  some 
means  of  legal  agreement,  through  the  release  of  special  design  documentation,  or  through  special 
characterization  tests.  Viitually  all  MILOTS  and  FME  and  Non-OTS  NDI  products  use  closed  archi¬ 
tectures  with  only  subsets  of  the  product  being  in  an  open  architecture.  Proprietary  and  market- 
driven  COTS,  ROTS,  and  FCOTS  products  are  inherently  closed  architectures,  requiring  extra 
actions  to  become  open  standards.  Only  those  designs  using  industry  standards,  widely  accepted 
market  standards,  government  coordination  standards,  or  international  standards  can  qualify  as  open 
architectures.  In  addition  to  being  available  to  the  public,  an  open  architecture  should  also  be  widely 
accepted  in  order  to  gain  the  desired  benefits  (i.e.,  cost  savings  throughout  the  life  cycle).  Wide 
acceptance  also  implies  strong  market  competition.  Low  acceptance  means  limited  market  competi¬ 
tion,  leading  to  potential  sole  source  procurements.  An  open  architecture  is  still  preferred  over  a 
closed  architecture,  even  in  a  low  market  acceptance  circumstance,  because  the  low  market  accep¬ 
tance  may  be  caused  by  recent  acceptance  of  the  standard  or  an  immature  market  waiting  for  users 
like  DoD.  But  multiple  standards  may  be  available  for  which  the  market  has  already  expressed  a 
preference;  select  the  architectural  standards  dictated  by  the  market  where  possible. 

An  open  architecture  is  sometimes  referred  to  as  a  “standards-based  architecture”  although  open 
architecture  is  a  broader  term  that  also  recognizes  market  standards,  not  merely  published  industry 
standards. 
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Open  architectures  are  much  preferred  in  order  to  utilize  NDI.  In  fact,  open  architectures  have 
characteristics  that  make  them  preferred  whether  NDI  is  used  or  not.  DoD  and  Navy  policy  is  to  use 
open  system  architecture,  with  various  agencies  even  defining  which  sets  of  standards  are  desired 
over  others  that  are  available  and  that  meet  the  criteria  of  being  open  standards.  Open  architectures 
have  three  primary  advantages:  (1)  flexibility  in  meeting  new  or  changing  requirements,  (2)  flexibil¬ 
ity  in  adopting  new  technologies,  and  (3)  lower  life-cycle  support  costs  (due  to  lower  modification 
costs).  The  flexibility  of  open  systems  can  be  a  major  advantage  when  using  NDI  because  support 
can  be  generated  over  the  long  term  even  when  suppliers  and  technologies  are  short-lived.  This  is 
especially  critical  for  COTS  products  in  emerging  technology,  highly  competitive  markets  populated 
by  small  businesses.  Open  architectures  can  also  be  used  to  define  product  requirements  that  are 
stable  even  when  the  operational  requirements  and  threats  associated  with  the  application  are  ill- 
defined  or  highly  dynamic. 

The  personal  computer  industry  provides  a  classic  example  of  the  effects  of  architecture.  The 
original  IBM  PC  was  an  open  architecture  because  IBM  published  the  information  for  interfacing  to 
its  BIOS  and  bus  structure  and  did  not  require  licensing;  the  Apple  Macintosh  was  a  closed  architec¬ 
ture  because  Apple  required  licenses  to  enforce  conformance  to  its  published  standards  and  not  all  of 
the  standards  were  published.  In  both  cases,  the  interface  design  was  company  unique  and  controlled 
by  the  original  design  activity,  characteristics  of  proprietary  standards.  The  IBM  standard  became  an 
open  standard  because  it  was  openly  published  and  quickly  adopted  as  a  market  standard.  The 
closed  architecture  allowed  Apple  to  control  quality,  user  interface,  upgrade  compatibilities,  and 
application  interoperability;  these  were  haphazard  features  on  the  IBM.  The  open  architecmre 
allowed  a  myriad  of  third-party  interface  products  and  application  products  to  become  available 
quickly.  This  had  the  immediate  effect  that  costs  dropped  quickly  for  IBM  standards-based  products, 
and  applications  meeting  newly  discovered  requirements  were  rapidly  developed.  Even  though  the 
early  days  of  the  PC  industry  had  many  start-up  companies  that  were  short-lived,  support  for  the 
open  architecture  products  became  much  cheaper  and  more  readily  available  than  for  similar  closed 
architecture  products.  Ultimately,  new  technology  products  became  available  for  the  open  architec¬ 
ture  much  more  rapidly  and  less  expensively  than  for  the  closed  architecture. 

The  more  normal  form  of  open  architecture  is  that  created  by  system  designers.  In  this  form,  the 
system  designers  perform  the  top-level  system  partitioning  using  industry  standards  as  prime  selec¬ 
tion  criteria  while  looking  ahead  to  the  potential  for  using  existing  products.  The  terminal  sets 
recently  specified  for  various  Navy  communications  requirements  serve  as  good  examples.  These 
architectures  are  all  very  similar.  Much  of  the  functionality  is  allocated  to  software  residing  on  a 
Navy  TAG  workstation.  The  TAG  workstation  program  uses  the  DoD  and  Navy  standards  for  open 
systems.  This  provides  a  stable  environment  for  the  acquisition  and  life-cycle  support  of  the  soft¬ 
ware  applications,  including  the  use  of  NDS.  The  form  and  fit  of  the  hardware  is  specified  to  con¬ 
form  to  industry/ISO-recognized  packaging  and  interface  standards,  such  as  VME  bus,  VXI  bus, 
19-inch  racks,  and  various  standard  LAN  standards.  This  approach  minimizes  the  proprietary  design 
features  when  they  do  exist  and  typically  isolates  the  proprietary  features  to  only  a  few  subassem¬ 
blies;  the  approach  also  promotes  commonality  with  commercial  communications  terminal  equip¬ 
ment  and  raises  the  potential  for  using  NDI,  especially  GOTS.  If  the  terminal  equipment  is  not  a 
GOTS  design,  it  is  often  integrated  NDI  where  most  of  the  integration  design  “glue”  is  implemented 
in  software.  (Proprietary  and  other  closed  architectures  may  often  be  converted  to  an  open  architec¬ 
ture,  if  sufficient  product  interface  information  is  available,  by  encapsulating  the  closed  architecture 
items  in  interfacing  products  that  translate  the  interfaces  into  an  open  architecture  set  of  standards.) 
Although  terminal  sets  with  this  architecture  may  have  high  initial  support  costs,  the  initial 
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acquisition  costs  and  life-cycle  support  costs  can  be  very  low.  The  sets  are  inherently  easy  to  recon¬ 
figure  to  new  communications  protocols,  easy  to  upgrade  with  new  technology,  and  cost  effective  to 
support  as  long  as  proper  logistics  decisions  are  put  into  place. 

INFLUENCES  OF  CURRENT  TECHNOLOGY 

Technology  advancements  are  constantly  influencing  how  system  partitioning  decisions  are  made. 
New  technologies  lead  to  new  interfacing  standards.  These  new  standards  usually  migrate  from 
being  company  proprietary  to  market-driven  to  industry  standards  over  time  as  the  technology 
becomes  more  widely  accepted.  Some  companies,  especially  Hewlett-Packard,  IBM,  and  AT&T, 
actively  pursue  establishing  new  technologies  that  they  have  developed  as  industry  standards.  These 
companies  participate  heavily  in  industry  standardization  activities.  While  the  company  risks  losing 
business  to  competitors  joining  in  the  industry  standard,  the  industry  standard  promotes  a  rapid 
incorporation  of  the  technology  into  a  variety  of  products  that  promotes  rapid  market  growth.  The 
rapid  market  growth  quickly  raises  production  volumes,  lowers  production  costs,  and  increases  the 
profitability  of  the  companies  positioned  to  take  advantage  of  the  changing  market  conditions. 

Often,  this  creates  more  than  one  standard  for  a  particular  function,  each  based  on  a  different 
technology.  It  may  also  create  a  hierarchy  of  standards  sometimes  called  an  architectural  suite  of 
standards.  In  an  architectural  suite,  a  series  of  tiers  or  layers  are  defined  to  connect  the  physical 
implementation  standards  to  the  application  or  user  standards.  The  intervening  layers  may  provide  a 
variety  of  different  standards,  usually  called  protocols,  so  that  selecting  a  standard  from  each  layer 
results  in  a  defined  capability.  Each  defined  capability  is  supported  by  a  stack  of  standards  called  a 
user  or  application  profile.  A  given  application  or  user  function  may  acmally  be  supportable  by  a 
vanety  of  profiles,  and  the  system  designer  usually  strives  to  define  profiles  across  all  of  the  system 
user  functions  that  have  the  least  variability  in  the  lowest  layers  of  the  stacks. 

This  approach  is  substantially  different  from  system  partitioning  before  1970.  Originally,  system 
partitioning  was  confined  to  the  best  means  of  implementing  a  system  function — in  mechanical, 
electrical,  or  electronic  hardware,  or  by  the  human  operator.  Software  implementations  started  to 
become  available  in  the  1950s,  but  the  functionality  options  were  severely  limited  by  technology.  In 
the  1980s,  the  price  of  computing  hardware  dropped  very  dramatically  and  software  technology  bar¬ 
riers  were  rapidly  tom  down,  rapidly  transforming  the  options  available  to  system  designers.  At  the 
same  time,  microelectronics  advances  created  the  new  design  options  of  programmable  hardware, 
production  tailorable  hardware,  and  cost-effective,  application-specific  integrated  circuits  (ASIC). 
The  number  of  options  available  to  a  system  designer  has  continued  to  grow  exponentially.  The 
number  of  solutions  leading  to  proprietary  standards  has  kept  pace  with  the  rapid  growth  of  industry 
standards,  so  the  challenge  to  the  system  designer  has  become  a  double  exponential  growth.  As  a 
result,  building  and  analyzing  user  profiles  is  but  one  of  the  ways  that  system  designers  can  organize 
and  understand  the  impact  of  the  many  competing  standards. 

The  capabilities  of  new  technology  have  led  to  the  development  of  a  generic  system  architecture. 
This  generic  architecture  consists  of  a  high  degree  of  functionality  allocated  to  software  running  on  a 
standard  computing  platform.  The  computing  platform  is  supported  by  ASIC  or  programmable^ hard¬ 
ware  technology  to  adapt  the  platform  to  specific  and  unique  application  environments  that  cannot  be 
readily  expressed  in  software.  Some  other  hardware  may  be  required  for  the  physically  expressed 
functionality  (such  as  engines  or  armor  or  munitions  or  antennae),  but  it  is  minimized.  Every  reason¬ 
able  effort  is  made  to  avoid  allocating  functions  to  human  operators.  Within  the  software,  the  func¬ 
tions  are  expressed  as  objects.  Just  as  the  functionality  breaks  down  into  less  complex  components, 
so  the  top-level  software  objects  can  be  assembled  from  lower  level  objects.  At  every  level,  there  are 
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a  multitude  of  options  for  selecting  different  types  of  standards  and  different  technology  standards  of 
each  type.  The  system  designer  needs  to  be  very  knowledgeable  in  the  standards  options  and  the 
associated  markets.  The  range  of  choices  can  be  mind-numbing,  but  it  also  increases  the  chances 
that  viable  NDI  products  or  standards  will  exist  to  satisfy  the  system  requirements. 

The  cleanest  way  to  express  software  functionality  in  a  system  architecture  is  to  partition  the  soft¬ 
ware  such  that  all  computer  software  configuration  items  (CSCI)  reside  on  a  single  processor.  This 
allows  easier  control  of  the  hardware/software  interfaces  and  promotes  testability  and  system  main¬ 
tainability  throughout  its  life  cycle.  However,  evolving  software  technology  has  defined  super¬ 
objects  that  can  bridge  or  adapt  to  changing  system  states  to  further  reduce  the  need  for  operator 
intervention.  These  super-objects  are  frequently  called  system  agents.  Software  agent  technology 
usually  results  in  expressing  the  agent  in  the  system  hierarchy  above  the  processors  hosting  the  soft¬ 
ware;  the  agent  is  truly  distributed  across  multiple  processors.  This  introduces  significant  challenges 
to  the  system  designers  and  life-cycle  agents  because  the  hardware/software  interfaces  must  be  care¬ 
fully  controlled  to  provide  a  stable  platform  for  the  agents  to  operate  properly.  However,  the  under¬ 
lying  processor  technology  is  subject  to  very  rapid  changes.  This  circumstance  requires  the  system 
designers  and  life-cycle  agents  to  maintain  a  very  detailed  configuration  status  of  both  the  hardware 
and  software  and  to  continuously  research  and  test  hardware  items  against  this  interface  in  order  to 
keep  the  interface  up  to  date  as  the  technology  changes. 

If  a  particular  function  finds  a  sufficient  market,  the  products  embodying  that  function  will  be  sub¬ 
jected  to  continuous  technology  insertion  and  product  update.  As  an  example,  disk  controllers  in 
early  personal  computers  were  discrete  component  designs  on  printed  wiring  boards  using  very  sim¬ 
ple  integrated  circuits.  Over  the  years,  the  function  has  been  sufficiently  standardized  and  integrated 
into  programmable  gate  arrays  and  application  specific  integrated  circuits,  allowing  the  function  to 
be  combined  with  many  other  functions  or  to  be  included  with  the  computer’s  motherboard.  Later 
generations  may  include  other  related  functions  such  as  caching  and  error  correction  and  compression. 
This  same  migration  of  functionality  from  assemblies  to  subassemblies  to  components  is  enabled  by 
the  combined  forces  of  market  economics  and  the  advances  of  current  technology. 

Generic  architectures,  software  technology  advances,  and  hardware  production  advances  each 
introduce  significant  challenges  to  maintain  a  system  configuration.  All  of  these  factors  are  usually 
at  work  in  modem  systems,  so  configuration  management  becomes  an  ever  greater  challenge,  espe¬ 
cially  for  NDI-based  systems.  These  challenges  imply  important  changes  in  the  way  configuration 
management  is  done,  but  the  new  problems  are  solvable.  See  CONFIGURATION  MANAGE¬ 
MENT  (CM)  ISSUES. 

MARKET  SURVEYS 

At  least  four  different  types  of  market  surveys  are  appropriate  for  the  different  phases  of  a  system 
life:  market  identification  survey,  initial  market  survey,  acquisition  market  survey,  and  system  sup¬ 
port  market  survey.  Each  of  these  survey  types  share  common  features,  so  it  is  often  possible  to  per¬ 
form  the  survey  functions  in  parallel.  The  market  identification  survey  gathers  information  to  iden¬ 
tify  what  market(s)  may  exist  to  support  a  particular  set  of  system  requirements  and  to  characterize 
the  economic  forces  and  quality  factors  driving  the  market(s).  The  initial  market  survey  identifies 
the  standards,  suppliers,  and  products  available  to  support  the  identified  system  technical  require¬ 
ments.  The  acquisition  market  survey  is  conducted  to  identify  the  suppliers  that  define  the  competi¬ 
tive  range  of  a  specific  acquisition  plan  or  contract  action  in  accordance  with  the  Federal  Acquisition 
Regulations  (FAR)  and  the  Defense  Federal  Acquisition  Regulations  (DFAR).  A  system  support 
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market  survey  gathers  information  about  the  stability  of  product  interfaces  against  system  require¬ 
ments  and  evaluates  repair  versus  replacement  versus  technical  upgrade  opportunities  through  the 
system  life-cycle  support  phase.  Each  of  the  surveys  contributes  valuable  information  for  making 
program  decisions,  and  the  survey  results  should  be  well  documented  in  a  form  readily  accessible  by 
the  program  decision  makers. 

A  market  identification  survey  is  most  efficiently  and  effectively  conducted  by  one  to  four  engi¬ 
neers  practicing  in  the  technology  and  keeping  up  to  date  in  their  field  through  trade  journals,  crnifer- 
ences,  short  courses,  and  so  forth,  supplemented  by  economic  data  on  the  suppliers  in  the  market.  If 
there  are  multiple  key  technologies,  multiple  survey  subgroups  of  engineers  can  be  used  to  gather  all 
of  the  information  needed.  The  engineers  involved  in  the  system  specification  and  design  are  usually 
well  qualified  for  the  survey  tasks. 

An  acquisition  market  survey  must  be  conducted  consistently  with  the  FAR  and  DEAR,  but  may 
limit  the  competitive  range  of  suppliers.  For  instance,  the  acquisition  market  survey  may  exclude 
vendors  that  are  not  currently  in  production  in  favor  of  suppliers  who  are  able  to  deliver  directly 
from  stock,  using  program  schedule  and  risk  criteria  as  justification.  Other  criteria  for  including  or 
excluding  suppliers  from  the  competitive  range  may  include  the  availability  of  field  data  or  tesUoc- 
umentation,  quality  criteria,  the  availability  of  suitable  support  services,  various  costs,  or  perfor¬ 
mance  factors.  A  sufficient  competitive  range  must  remain  to  ensure  adequate  competition.  The 
quality  of  the  survey  documentation  can  often  be  critical  to  avoiding  protests  in  subsequent  contract¬ 
ing  actions. 

Initial  Market  Survey 

The  initial  market  survey  should  be  considered  a  mandatory  part  of  the  system  design  process  of 
establishing  system  partitions.  For  NDI  acquisitions,  the  survey  continues  into  the  procurement 
phase,  and  should  be  transitioned  into  a  system  support  market  survey  for  the  support  of  the  system 
(in  order  to  make  effective  item  replacement  decisions).  For  developmental  acquisitions,  the  survey 
is  handed  over  to  the  design  engineer,  who  continues  a  mini-survey  on  the  assigned  portion  of  the 
development.  The  initial  market  survey  consists  of  the  following  activities; 

1 .  Surveying  existing  standards  that  address  functional  requirements. 

2.  Surveying  potential  suppliers  and  the  market  economic  and  quality  factors. 

3.  Surveying  existing  products  and  technologies. 

4.  Obtaining  additional  information  from  suppliers  and  users  of  candidate  technologies,  espe¬ 
cially  cost  data. 

5.  Obtaining  additional  information  from  suppliers  and  users  of  candidate  products,  including  test 
data,  quality  data,  reliability  data,  usage  data,  cost  data,  supportability  data,  and  environmental 
data. 

6.  Developing  a  tailored  screen  to  supplement  information  not  completed  otherwise  and  to  deter¬ 
mine  the  viable  candidates. 

7.  Adjusting  requirements  through  industry  comments  on  procurement  requirements  documents 
(specifications,  statements  of  work,  and  contract  data  requirements  lists). 

The  initial  market  survey  through  step  5  should  be  a  part  of  every  acquisition,  including  new  devel¬ 
opments.  The  survey  should  incorporate  expertise  covering  reliability,  test,  quality,  integrated 
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logistics,  and  documentation  as  well  as  the  functional  technology  expertise.  This  expertise  should  be 
combined  to  characterize  the  technology  and  economic  factors  that  drive  the  product  market.  Even  if 
an  off-the-shelf  option  is  not  eventually  selected,  this  market  characterization  can  be  used  by  the  sys¬ 
tem  designer  to  select  a  viable  architecture  and  appropriate  standards  to  minimize  the  system  life- 
cycle  support  costs.  The  system  designer  should  ensure  that  the  results  are  documented  for  use  dur¬ 
ing  the  procurement  phases  that  will  follow  the  system  design.  NDI  introduces  issues  in  each  of 
these  areas. 

In  addition  to  evaluating  technology  factors  against  requirements,  the  initial  market  survey  should 
also  evaluate  cost  factors.  This  is  especially  important  for  high-technology  items  and  rapidly  chang¬ 
ing  technologies.  In  general,  product  costs  are  high  when  the  product  is  first  introduced  on  the  lead¬ 
ing  edge  of  the  technology  market.  Prices  come  down  as  the  product  matures  and  the  market 
demand  supports  high  production  rates  and  the  investment  in  manufacturing  efficiencies.  As  the 
technology  matures,  new  products  are  introduced  that  take  advantage  of  the  latest  innovations;  these 
products  often  go  beyond  a  mere  product  improvement  and  usually  establish  new  standards  for  inter¬ 
face  as  well  as  performance.  It  is  important  to  ascertain  where  a  product  is  in  its  life  cycle,  and  the 
cost  factors  provide  a  substantial  insight  into  the  product  maturity.  Figure  1  shows  a  typical  product 
life-cycle  cost  curve.  Figure  2  shows  typical  relative  costs  for  competing  technologies  on  the  open 
market. 

Figure  1  only  refers  to  the  product  cost  off-the-shelf  over  the  product’s  life  cycle  on  the  market.  It 
does  not  refer  to  the  costs  that  will  be  incurred  to  support  the  product  in  the  field.  However,  support 
costs  for  an  item  will  also  follow  a  similarly  shaped  curve  since  the  support  items  are  produced  on 
the  same  production  lines.  After  the  production  line  is  closed  down,  support  items  are  provided  from 
inventory.  Market  costs  tend  to  rise  because  inventory  management  costs  must  be  recovered.  Also, 
there  may  be  a  residual  demand  competing  for  the  increasingly  scarce  product,  combined  with 
diminishing  sources  of  supply  as  companies  shift  their  production  assets  to  the  newer  product  lines. 
Eventually,  the  shelf  inventory  will  be  used  up,  and  true  parts  can  only  be  obtained  through  the 
surplus  market  or  from  a  reopened  limited  production  line.  Clearly,  the  surplus  route  introduces 


Figure  1 .  Typical  product  shelf  cost  over  its  life  cycle. 
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Figure  2.  Relative  costs  of  competing  technologies. 

huge  quality  risks  since  items  may  have  been  scavenged  from  equipments  taken  out  of  service  for 
unknown  reasons.  Opening  a  limited  production  line  is  very  expensive.  Quality  on  limited  produc¬ 
tion  runs  is  often  very  difficult  to  maintain,  representing  a  significant  risk.  Clearly,  the  choice  is  to 
avoid  both  surplus  and  limited  production  situations. 

Figure  2  illustrates  the  typical  relative  costs,  citing  the  example  of  hard  disk  drives  for  personal 
computers  in  the  late  1995  time  frame.  The  10-  to  40-MB  hard  disks  primarily  represent  MFM  inter¬ 
face  technology.  These  disks  were  not  only  smaller  in  capacity,  but  larger  in  size  and  weight  and 
much  slower  in  access  times.  However,  they  were  still  available  in  the  surplus  market  for  about  $1  a 
piece,  although  most  vendors  could  not  guarantee  their  operation.  The  100-  to  500-MB  drives  were 
primarily  early  IDE  drives.  Although  most  vendors  did  not  stock  this  size  drive  in  late  1995,  prices 
were  in  the  $80  to  $160  range.  The  800-  to  1600-MB  drives  represented  the  mature  edge  of  the 
technology  in  late  1995,  with  prices  running  from  $180  to  $275  (plus  some  variance  between  ven¬ 
dors).  A  1.6-GB,  extended  IDE  Mode  4  drive  was  the  buy  of  late  1995.  Drives  of  significantly 
higher  capacity  were  typically  SCSI  technology  and  were  available  at  a  somewhat  higher  cost  per 
MB  of  storage,  but  had  very  fast  access  times.  Also,  various  optical  storage  technologies  started 
becoming  price  competitive  in  1995,  but  access  times  and  transfer  rates  were  still  behind  hard  disk 
technology.  The  relative  costs  represented  by  these  numbers  are  very  typical  across  any  technology 
that  has  a  significant  market  with  a  lot  of  market  growth  as  well  as  high  levels  of  technical  innova^ 
tion.  The  ideal  buy  point  is  highly  time  sensitive  and  the  performance  represented  at  this  level 
constantly  changing.  Markets  without  these  levels  of  growth/innovation  will  have  similar  cost 
curves,  but  the  time  represented  from  edge  to  edge  will  be  much  longer  than  for  the  truly  high- 
technology  markets. 

It  is  important  to  choose  a  correct  metric  model  in  evaluating  price  and  performance  in  the  market. 
Take  the  hard  disk  drive  example.  A  "price  only”  model  will  dictate  buying  something  oIT  of  the 
surplus  market,  so  a  40-MB  MFM  drive  for  $1  is  likely  to  be  chosen.  A  “performance  only”  model 
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will  likely  drive  the  selection  toward  a  90-GB  SCSI  drive  for  $200,000.  A  “comprehensive”  model 
properly  balancing  a  variety  of  performance  factors  with  price  will  likely  choose  a  large,  fast 
extended  IDE  or  SCSI  drive,  depending  on  the  performance  factors  that  are  important.  In  late  1995, 
the  likely  choice  would  have  been  an  extended  IDE  drive  with  a  1.6-GB  capacity  and  9-  to  10-ms 
access  time  for  about  $240.  Figure  3  illustrates  these  choices. 


Figure  3.  Metric  models  in  market  evaluations.  Different  metrics  produce  dramaticaliy  different 
decision  points  for  price  and  performance  to  meet  requirements. 

Different  acquisition  programs  will  have  different  priorities,  so  the  metric  models  for  evaluating 
the  market  should  reflect  these  differences.  There  may  or  may  not  be  a  common  solution  for  several 
projects  containing  similar  functional  requirements.  Selections  should  initially  be  based  on  the 
immediate  acquisition  project  requirements.  After  a  workable  system  architecture  has  been  selected, 
then  alternatives  influenced  by  commonalities  with  other  systems  should  be  considered.  The  system 
partitioning  decisions  should  be  standards  driven.  However,  the  standards  may  reflect  the  rapidly 
developing  technology  within  a  market,  industry  standards,  or  previously  adopted  standards  already 
supported  for  other  projects.  Large  acquisitions  should  tend  toward  the  latest  market  standards  while 
very  small  acquisitions  should  tend  toward  previously  adopted  standards.  These  are  the  decisions 
that  will  be  reached  as  a  result  of  a  life-cycle  cost  analysis.  Small  acquisitions  that  share  standards 
with  other  projects  should  share  the  life-cycle  management  resources  of  those  other  projects  as  well 
to  obtain  the  benefits  of  mutual  upgrades.  See  SYSTEM  LIFE-CYCLE  PLANNING. 

System  Support  Market  Survey 

System  support  market  surveys  are  very  similar  to  initial  market  surveys,  and  the  initial  survey  can 
be  flowed  into  the  system  support  survey  for  NDI  acquisitions.  Although  system  support  market  sur¬ 
veys  may  be  conducted  occasionally  as  part  of  a  product  improvement  acquisition  or  when  spare  part 
availability  problems  arise  for  developed  systems,  they  must  be  conducted  virtually  continuously  for 
systems  employing  high-technology,  off-the-shelf  products.  The  system  support  market  survey  con¬ 
sists  of  the  following  activities: 
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1.  Surveying  existing  interface  standards  associated  with  the  system  support  products,  including 
the  market  economic  and  quality  factors  and  the  incremental  product  changes  (in  relation  to  ° 
these  standards). 

2.  Surveying  new  related  products  and  technologies,  especially  cost  data. 

3.  Obtaining  additional  information  from  suppliers  and  users  of  candidate  products,  including  test 

data,  quality  data,  reliability  data,  usage  data,  cost  data,  supportability  data,  and  environmental 
data. 

4.  Developing  a  tailored  screen  to  supplement  information  not  completed  otherwise  and  to  deter¬ 
mine  the  viable  candidates. 

The  system  support  market  survey  is  limited  to  the  identified  level  of  repair  for  the  system  (such  as 
for  a  specific  VME  card  or  software  product),  unlike  the  initial  market  survey  that  is  applied  itera- 
hvely  to  the  many  levels  in  the  system  partitions.  In  practice,  most  of  the  survey  effort  is  embodied 
in  obtaining  test  items  in  order  to  screen  minor  modifications  in  the  products  that  make  up  the  sys¬ 
tem,  especially  in  those  products  that  are  available  from  multiple  suppliers.  This  effort  can  be 
limited,  but  there  needs  to  be  sufficient  effort  so  that  product  interfaces  (especially  hardware- 
software  interfaces)  can  be  evaluated  ahead  of  having  to  make  repair,  replace,  or  upgrade  decisions. 


TAILORED  SCREENS  FOR  NDI 

Screens  are  special  investigations  and  tests  conducted  to  identify  and  certify  products  are  viable 
for  application  to  a  defined  set  of  requirements.  The  basis  for  the  screen  is  always  the  well-defined 
requirements  established  early  in  the  acquisition  program.  Ideally,  these  requirements  are  complete, 
stable,  clear,  concise,  measurable,  and  stated  in  terms  that  can  be  applied  directly  to  the  system 
design  problem.  Seldom  are  requirements  so  well  behaved.  Usually,  requirements  are  incomplete 
in  some  major  areas  (especially  support  and  human  factors  requirements),  subject  to  substantial 
changes,  stated  in  terms  that  are  incomprehensible,  unmeasurable,  and  difficult  to  inteipret  in  system 
design  terms.  Prior  to  proceeding  with  system  partitioning  actions,  it  is  necessary  to  conduct  detailed 
requirements  analyses  to  resolve  the  differences  between  the  ideal  and  the  reality.  In  the  process  of 
conducting  the  requirements  analyses,  it  is  helpful  to  also  conduct  investigations  into  the  markets 
that  could  provide  potential  NDI  products.  Even  if  no  viable  products  exist,  these  investigations 
establish  a  baseline  for  what  constitutes  the  state  of  the  art  for  the  operational  requirements;  this 
information  is  needed  to  write  legally  viable  contract  specifications  and  aids  in  conducting  the  initial 
market  surveys  during  the  system  partitioning  phase  of  system  design. 

Having  established  the  requirements,  the  system  designer  proceeds  with  the  system  partitioning 
decisions.  At  each  level  of  complexity,  the  appropriate  standards  must  be  identified.  In  most  cases, 
these  standards  are  embodied  in  existing  products,  so  a  search  of  the  existing  markets  for  products  is 
a  quick  means  for  gathering  the  desired  standards  information.  The  existence  of  NDI  greatly  pro¬ 
motes  system  design.  Furthermore,  the  NDI  can  be  used  to  form  a  baseline  comparison  system.  The 
baseline  comparison  system  represents  all  of  the  required  system  functionality  as  expressed  in  the 
existing  products.  The  NDI  included  in  the  comparison  system  does  not  have  to  include  viable  can¬ 
didate  NDI  for  inclusion  in  the  system  under  design,  merely  to  represent  similar  functionality.  Such 
a  comparison  system  is  extremely  useful  in  performing  logistics  analyses  and  in  defining  the  nonper¬ 
formance  systems  requirements.  It  is  unlikely  that  any  part  of  the  new  state-of-the-art  system  bein® 
acquired  will  have  many  of  the  product  elements  of  the  baseline  comparison  system;  nevertheless,  ° 
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the  actual  field  data  gathered  for  the  baseline  comparison  system  is  significantly  more  reliable  data 
for  making  support  decisions  than  the  models  and  forecasts  arising  from  pure  analyses.  Using  NDI 
in  this  way  naturally  exposes  potential  viable  candidates  for  inclusion  in  the  new  system. 

As  the  system  partitioning  for  the  first  two  of  three  levels  of  complexity  is  completed,  a  host  of 
high-level  products  and  standards  will  be  identified.  Also,  the  top-level  requirements  should  be  well 
defined  at  this  stage.  A  further  market  analysis  looking  ahead  to  the  lower  levels  of  complexity  will 
further  identify  candidate  NDI.  However,  a  large  number  of  the  NDI  candidates  are  likely  to  be 
designed  against  proprietary  standards  or  market  standards  that  are  undefined — unknowable  to  the 
system  designer  or  so  highly  variable  as  to  present  a  significant  system  design  risk.  For  the  various 
forms  of  MILOTS.  the  system  designer  needs  to  check  the  specifications  and  test  reports  for  the 
items  to  see  if  there  are  any  deviations  or  waivers  on  requirements  or  other  nonconformances  to 
requirements  that  are  critical  to  the  new  design  and  also  to  determine  the  technical  assumptions  and 
conditions  of  the  specification  or  report.  For  items  in  production,  it  is  useful  to  check  production 
quality  records  to  see  what  the  defect  rates  are  and  to  characterize  the  nature  of  the  defects.  Many 
sources  of  supply  are  required  to  keep  such  records,  and  the  records  can  be  disclosed  under  IS09000 
(and  MIL-Q-9858)  quality  provisions.  Field  data  should  be  available  for  all  forms  of  _OTS,  except 
NewCOTS.  All  of  this  data  allows  the  system  designer  to  narrow  the  field  of  likely  candidates. 

The  field  of  likely  candidates  can  be  narrowed  even  more  by  eliminating  those  that  do  not  meet  the 
hard  technical  requirements  and  those  that  cannot  conform  to  the  system  acquisition  constraints 
(such  as  cost  or  support  requirements).  The  remaining  candidates  will  include  those  that  are  truly 
off-the-shelf  and  tlu>se  that  are  not.  The  candidates  that  are  not  off-the-shelf  may  include  many 
NewCOTS  and  some  military  equipments  that  are  not  in  production  and  that  are  available  only  after 
a  sufficient  quantity  is  needed  (then  the  equipment  must  be  “redeveloped”  for  each  new  production 
run).  Of  all  of  the  remaining  candidates,  it  is  unlikely  that  any  will  meet  all  of  the  requirements 
flowed  down  to  the  product  level  they  represent.  Each  candidate  must  be  assessed  for  the  ability  to 
modify  or  to  isolate  the  candidate  in  such  a  way  that  the  nonconformances  can  be  accommodated  in 
the  system  design  For  instance,  a  particular  candidate  might  not  be  capable  of  withstanding  the 
severe  shock  and  \  ihratirm  of  a  shipboard  environment,  but  a  suitable  installation  kit  can  provide  suf¬ 
ficient  isolation  for  the  candidate  to  remain  viable.  An  off-the-shelf  software  package  might  not 
include  a  particular  interlace  function,  but  an  included  macro  language  capability  might  allow  the 
functionality  to  be  added  in  easily.  An  item  might  have  unsuitable  human  factors,  but  the  item  can 
be  hidden  behind  a  shell  that  does  provide  the  required  human  interface.  An  item  may  have  a  highly 
proprietary  design,  but  support  can  be  provided  so  that  repair  of  the  item  is  not  needed  above  the 
depot  level.  In  each  case,  the  costs  of  the  adaptation,  modification,  or  other  accommodation  must  be 
assessed.  This  includes  the  acquisition  and  the  support  costs.  In  the  end,  there  will  likely  remain  a 
number  of  potenliall)  viable  candidates,  and  a  number  of  risk  areas  will  be  identified  where  the  can¬ 
didate’s  performance  is  either  variable  or  unknown. 

The  potential  candidates  can  be  prioritized  based  on  the  conformance  to  the  system  requirements, 
their  success  as  products,  and  their  overall  suitability.  (Suitability  might  be  indicated  by  high  field 
reliability  experience,  inexpensive  long-term  warranties,  specified  ruggedness,  high  customer  satis¬ 
faction,  or  high  market  share.)  A  screen  can  now  be  applied  against  the  requirements  to  determine 
the  product  viability  within  the  proposed  system  design.  The  screen  should  consist  of  the  following 
elements;  visual  inspections,  data  reviews,  and  tests.  Application  of  the  screen  will  eliminate  some 
candidates  and  validate  others.  Those  that  are  validated  can  also  be  rated  technically  for  contracting 
purposes,  based  on  the  screen  results  and  published  contract  evaluation  criteria.  (See  SOLICITA¬ 
TION  AND  SOURCE  SELECTION  TIPS.) 
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Candidate  equipment  should  be  inspected  for  at  least  the  following  attributes:  workmanship, 
enclosure  effectiveness,  human  engineering,  safety  design,  maintainability,  operability,  thoughtful¬ 
ness  of  design  (functional  design  integrity,  flexibility  for  adaptation  or  modification,  and  producibil- 
ity),  and  vintage  of  design.  Major  and  minor  discrepancies  should  be  noted.  Major  discrepancies  are 
cause  for  rejection  if  they  cannot  be  corrected  through  simple  modifications.  An  archaic  design  may 
be  cause  for  rejection  unless  the  supplier  can  guarantee  support  for  a  time  compatible  with  the 
required  product  life.  A  design  that  is  too  new  (experimental  or  unproved)  may  inject  significant 
risks  that  must  be  addressed  in  the  overall  acquisition  strategy. 

The  data  review  should  include  all  available  specifications,  test  reports,  technical  manuals,  instruc¬ 
tions,  schematics,  procedures,  and  installation  data.  These  data  items  should  be  reviewed  for  consis¬ 
tency  and  completeness,  comprehensibility,  and  utility.  For  hardware  items,  the  following  elements 
are  useful  indicators: 

•  Net  input  power/volume  (an  indication  of  heat  buildup  and  reliability). 

•  Operating  temperature  (an  indication  of  component  design  limitations  and  humidity  resis¬ 
tance). 

•  Cooling  capability/net  input  power  (an  indication  of  cooling  effectiveness). 

•  Internal  voltage  levels  (another  indication  of  humidity  resistance  and  potential  safety  prob¬ 
lems). 

•  Equipment  weight,  volume,  shape,  and  mounting  provisions  (indicators  of  shock  and  vibration 
resistance,  and  interface  or  installation  problems). 

•  Manufacturer  s  claimed  environmental,  reliability,  and  interface  specification  conformance,  if 
any. 

•  Component  weight,  volume,  shape,  and  mounting  provisions  (further  indicators  of  vibration 
and  shock  resistance). 

•  Electronic  and  mechanical  component  counts  (indicators  of  reliability,  maintainability,  and 
cost  of  logistics  support). 

These  data  items  should  be  consistent  with  each  other  and  with  supplier  claims.  Inconsistencies  indi¬ 
cate  risks  and  may  expose  unreliable  data  that  should  not  be  used  in  system  planning.  Incomplete 
data  may  be  an  indication  of  maintaining  security  for  proprietary  data,  tuning  data  to  look  better  than 
it  should,  sloppiness  in  the  data  preparation,  or  honest  omissions;  the  supplier  may  also  be  lying. 
Comparative  data  between  products  may  use  industry  benchmarks  but  be  based  on  “apples  and 
oranges  tests,  such  as  comparing  the  SPECint92/SPECint92fp  performance  between  competing 
computers  but  including  secondary  cache  on  one  but  not  the  other.  Other  data  may  be  gathered 
under  unknown  conditions,  so  the  utility  for  making  decisions  is  limited.  Some  commercial  markets 
simply  do  not  require  many  types  of  documentation  common  to  military  systems.  In  any  case, 
incompleteness  also  indicates  risks.  Where  documentation  is  not  available,  the  information  still 
needs  to  be  developed.  The  last  resort  is  the  screening  test. 

For  software,  the  screen  is  equivalent  in  form,  but  the  issues  are  different.  The  same  basic  data 
elements  are  gathered  from  the  same  types  of  sources  (specifications,  test  reports,  quality  data,  etc.) 
but  the  form  of  the  data  and  expertise  needed  to  interpret  the  data  is  different  from  hardware  prod¬ 
ucts.  The  software  must  reside  in  a  hardware  environment,  and  comparative  data  from  different 
sources  usually  do  not  provide  information  from  the  same  environment.  Even  if  they  do,  the  data  is 
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probably  for  an  environment  substantially  different  from  that  contemplated  by  the  system  designers. 
Third-party  evaluations  can  also  contain  unknown,  unintended  biases.  For  instance,  independent 
evaluations  of  three  popular  word  processors  by  six  different  sources  over  a  4-month  period  resulted 
in  six  different  rank-orders — all  of  the  possible  results.  On  the  surface,  each  evaluation  was  looking 
at  the  same  factors  and  using  a  common  methodology.  In  practice,  the  results  were  being  skewed  by 
the  different  test  tasks  and  the  familiarity  of  the  evaluators  with  one  approach  versus  another. 

Another  software  issue  arises  from  the  sharing  of  hardware  resources  with  other  software  pro¬ 
cesses.  Conflicts  between  applications  can  and  do  arise  and  can  result  in  acceptable  degradation  of 
the  speed  of  execution  to  system  crashes.  Most  of  these  type  of  issues  cannot  be  resolved  by  market 
survey  and  literature  search  results;  they  can  only  be  resolved  through  testing. 

Screening  tests  for  software  are  essential,  but  they  can  range  from  simple  demonstrations  to  com¬ 
plex  tests  similar  in  scope  to  full  qualification  tests.  In  any  case,  a  common  set  of  metrics  should  be 
used  for  the  evaluations  that  are  tailored  to  the  intended  applications  and  consistent  throughout  the 
system  environment.  The  metrics  should  be  stated  in  “application  process  units”  (APU)  that  describe 
real  processes  functionally  present  in  the  system.  SPECint92  and  similar  benchmark  systems  are 
defined  at  a  lower  level  of  functionality.  APU  measures  should  be  defined  in  ways  meaningful  to  the 
system  users.  While  it  may  require  considerable  effort  to  define  APU  metrics,  they  can  be  used 
throughout  the  system  life  and  are  highly  beneficial  to  the  field  user  when  upgrades  are  provided 
with  APU  benchmarks.  Software  screens  should  be  done  in  a  common  hardware  environment — 
ideally  the  same  environment  that  will  be  present  in  the  fielded  system.  When  the  hardware  plat¬ 
forms  are  still  undergoing  selection,  a  common  test  environment  that  closely  simulates  the  expected 
system  can  be  acceptable. 

The  screening  test  is  tailored  to  develop  information  that  is  otherwise  missing  and  to  validate  that 
critical  requirements  can  be  met.  Samples  of  the  candidate  NDI  surviving  to  this  stage  are  obtained 
for  testing.  At  least  three  samples  of  each  candidate  is  desired,  although  one  is  often  all  that  can  be 
obtained.  Fewer  than  three  test  samples  simply  increases  the  risks  associated  with  the  validation 
function,  so  an  actual  “certificate  of  validation”  may  be  delayed  until  after  the  system  is  in  produc¬ 
tion.  In  any  case,  the  tests  should  include  the  following  steps  for  hardware  and  hardware/software 
products: 

1.  Initial  performance  tests  to  the  manufacturer’s  specifications;  where  several  parameters  inter¬ 
act,  choose  “worst-case”  test  conditions. 

2.  Testing  to  project  specifications,  including  environmental  limits. 

3.  Bum-in  for  100  clock  hours  (on  each  item). 

4.  Combined  environment  stress  test  including  temperamre,  humidity,  vibration,  and  electrical 
transients  plus  stresses  from  environmental  cycling,  EMI,  and  repetitive  shock,  as  appropriate. 

The  combined  environment  test  normally  requires  120  hours.  The  entire  test  cycle  can  be  completed 
in  3  weeks.  With  suitable  environmental  test  equipment,  all  of  the  candidates  can  be  done  in  parallel. 
Of  course,  the  total  calendar  time  required  may  be  extended  by  excessive  failures,  by  repair  time,  or 
by  facility  scheduling  problems.  The  screening  acceptance  criteria  are  as  follows; 
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1 .  No  repeatable  failures  for  any  single  test  sequence.  (For  example,  failure  of  a  vibration  test 
twice  is  not  acceptable.) 

2.  No  pattern  failures  (two  or  more  failures  of  the  same  part  in  equivalent  applications  caused  by 
the  same  failure  mechanism.)  Passing  requires  failure-free  performance  through  the  combined 
environment  testing  phase  on  at  least  one  out  of  three  trials;  high-reliability  items  should  pass 
in  a  single  trial. 

3.  Measured  MTBF  meets  or  exceeds  project  requirements  with  acceptable  confidence. 

4.  Predicted  performance  remains  within  required  limits  when  weighed  by  quality  factors. 

5.  Supplier  claims  for  performance  and  quality  are  verified  within  acceptable  limits.  (Large  vari¬ 
ations  between  products  and  deviations  from  supplier  claims  make  the  overall  product  quality 
questionable.) 

6.  Overall  conformance  to  program  requirements. 

Ideally,  all  of  the  candidates  will  be  qualified  without  reservation.  Usually,  some  candidates  will  be 
eliminated,  and  others  will  be  identified  as  requiring  modification.  The  candidates  passed  without 
reservation  plus  those  candidates  that  require  only  minor  modifications  constitute  the  surviving  pool 
of  candidate  NDl  for  use  m  the  system.  If  there  are  no  survivors,  proceed  with  the  system  partition¬ 
ing  for  another  tu  o  lev  els  of  complexity  and  repeat  the  process.  The  additional  system  design 
exposes  lower  lev  el  standards  and  their  associated  NDI  representatives.  Generally,  the  entire 
screening  process  is  conducted  in  concert  with  the  system  partitioning  activities  and  does  not  add  to 
the  overall  project  schedule.  However,  some  systems  may  require  substantial  amounts  of  testing, 
which  must  be  planned  and  funded.  Nevertheless,  the  exercise  of  screening  tests  develops  valuable 
information  that  is  usef  ul  for  designers  even  when  no  candidates  survive.  Validating  NDI  candidates 
can  greatly  shorten  the  overall  acquisition  cycle  and  provide  substantial  cost  savings  that  exceed  the 
costs  of  screening  h\  many  orders  of  magnitude. 

There  are  several  times  when  screening  tests  might  be  conducted.  The  tailored  screen  and  the 
market  analysis  are  intimately  linked,  so  this  is  often  a  time  for  conducting  some  tests  and  occurs 
prior  to  entering  a  procurement  decision.  Test  samples  for  these  tests  must  be  procured  or  obtained 
via  some  access  agreement  such  as  a  Cooperative  Research  And  Development  Agreement 
(CRADA).  Once  a  pri>curement  decision  has  been  reached,  tailored  screen  tests  are  often  used  to  fill 
in  information  gapN  needed  to  continue  system  planning.  These  tests  may  be  conducted  as  part  of  the 
procurement  and  mav  include  contractor  demonstrations.  Tests  may  also  be  required  after  system 
fielding  to  evaluate  product  improvements;  test  samples  can  be  required  as  part  of  the  initial  procure¬ 
ment  or  through  the  CR.-\D.V  vehicle. 

Screening  processe.s  are  applied  for  the  initial  acquisition  of  systems  or  of  system  components.  An 
often-neglected  acquisition  issue  is  the  need  to  acquire  replacement  spares  and  to  certify  improved 
products— hardware,  software,  and  supporting  training  products.  Since  screening  processes  empha¬ 
size  interface  requirements  and  specifications,  tailored  screens  can  be  employed  throughout  the  sys¬ 
tem  life  to  maintain  the  integrity  of  the  system  components.  It  is  often  necessary  to  maintain  some 
form  of  market  surveillance  for  potential  spares  throughout  the  system  life.  The  market  surveillance, 
spares  screening,  and  configuration  status  accounting  efforts  must  interact  with  each  other  and 
should  be  coordinated  by  the  same  organization — usually  the  Life-Cycle  Management  Activity. 

Throughout  the  screening  process,  and  especially  in  conjunction  with  the  market  analyses  and  sur¬ 
veys,  it  is  essential  to  incorporate  the  best  technical  expertise  available.  This  technical  expertise 
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must  be  knowledgeable  in  the  state  of  the  art,  in  the  market  players,  the  technology  players,  the 
evolving  standards,  and  the  directions  technology  is  likely  to  move.  Not  surprisingly,  one  individual 
can  seldom  cover  all  of  these  issues — a  team  must  usually  be  assembled.  It  is  also  important  to 
incorporate  good  operational  expertise  to  help  to  resolve  the  user-maintainer  requirements.  The  gov¬ 
ernment  laboratory  is  chartered  to  be  the  prime  resource  of  this  “smart  buyer”  expertise. 

Technical  Document  108  provides  additional  information  on  Screening  Techniques  and  making 
Build,  Buy,  or  Modify  decisions.  The  screens  should  be  structured  to  be  conducted  quickly  (1  to  3 
months)  in  order  to  ensure  relevance  of  the  screen  results  in  subsequent  procurements. 


CONTRACTING  FOR  NDI 

NDI  should  be  procured  using  fixed-price  contracts.  Since  the  items  already  exist,  there  is  no 
developmental  risk.  Even  for  Integrated  NDI  or  Major  Modifications  of  NDI,  the  scope  of  the  efforts 
for  integration  or  modification  are  limited.  An  acquisition  that  has  so  much  risk  that  a  fixed-price 
contract  should  not  be  used  is  also  beyond  the  scope  of  the  definitions  of  NDI. 

Fixed-price  contracting  should  be  characterized  by  several  features  in  contrast  to  the  cost  contracts 
normally  used  for  the  acquisition  of  items  requiring  development.  These  features  include  the  follow¬ 
ing: 

•  Well-defined  product  requirements. 

•  Stable  product  requirements. 

•  Well-defined  evaluation  and  acceptance  criteria. 

•  Well-defined  test  criteria. 

•  Well-defined  support  criteria. 

•  Acceptably  low  risk  to  both  the  acquirer  and  the  supplier. 

Notice  the  common  theme  in  these  characterizations — ^WELL  DEFINED.  It  may  require  consider¬ 
able  effort  to  make  an  acquisition  well  defined,  especially  when  system  requirements  are  fluid. 

Special  issues  arise  when  contracting  with  foreign  entities.  These  may  include  rights  to  data, 
translation  of  documentation  into  English,  accessibility  to  spares  and  reprocurement  rights,  availabil¬ 
ity  of  contractor  technical  services,  and  rights  to  product  improvements.  There  may  also  be  ques¬ 
tions  about  the  proper  interpretation  of  requirements,  especially  those  that  are  hard  to  translate.  This 
may  also  imply  special  testing  that  would  not  be  required  of  domestic  sources  of  supply.  Many 
times,  these  issues  are  resolved  by  the  foreign  bidder  by  teaming  with  a  domestic  source.  However, 
even  when  a  domestic  source  is  involved,  the  acquiring  agency  must  verify  that  these  issues  are 
resolved. 

The  structure  and  legal  ramifications  of  fixed-price  contracts  require  that  contract  specifications  be 
feasible  and  that  the  statement  of  work  requirements  be  so  well  defined  that  independent  estimates 
will  agree  closely  with  bidder  estimates.  The  item’s  support  should  be  acquired  at  the  same  time; 
therefore  the  support  requirements  should  be  well  integrated  into  the  specifications  and  SOW.  Often, 
the  item’s  support  will  take  advantage  of  commercial  warranties;  however,  there  may  be  special 
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requirements  that  must  be  implemented  to  keep  the  commercial  warranty  valid.  Each  of  the  different 
types  of  NDI  has  special  or  peculiar  elements  that  need  to  be  incorporated  into  the  contract  require¬ 
ments  in  some  way.  These  issues  are  discussed  in  the  sections  that  follow. 


WRITING  SPECIFICATIONS 

Specifications  provide  the  means  of  documenting  and  communicating  technical  requirements. 
There  are  many  different  kinds  of  technical  requirements,  just  as  there  are  many  levels  of  complexity 
m  any  system.  Reasons  for  documenting  and  communicating  requirements  include  procurement, 
guiding  development,  transition  of  responsibilities  between  agencies  (as  between  a  developer  and  a 
Software  Support  Activity  or  In-service  Engineering  Agent),  documenting  interfaces  for  potential 
system  upgrades  or  modifications,  and  documenting  information  for  future  use  by  users,  maintainers, 
and  repair  agents.  Each  of  these  purposes  tends  to  produce  specifications  with  their  own  special 
requirements  for  the  writer.  Specifications  for  NDI  acquisitions  do  not  differ  significantly  from 
developmental  acquisitions  except  for  procurements.  NDI  procurement  specifications  have  special 
charactenstics  to  satisfy  contracting  needs,  market  needs,  system  design  needs,  and  future  support 
needs. 

System  specifications  are  often  vague  with  many  “To  Be  Determined”  (TBD)  requirements  in 
some  sections  while  being  overly  rigorous  in  others.  This  is  a  natural  (and  acceptable)  element  of 
developmental  acquisitions,  but  it  is  unacceptable  for  procuring  NDI.  Developmental  efforts  will 
include  efforts  to  define  requirements  sufficiently  in  the  SOW;  these  efforts  must  precede  an  NDI 
acquisition  effort  or  be  included  in  the  contracting  plan.  In  fact,  the  thrust  of  the  specification  for 
NDI  differs  significantly  from  that  of  a  development  or  from  the  system  specifications.  Many  of  the 
requirements  for  NDI  specifications  are  consistent  with  DoD  policies  for  performance-based 
specifications  and  for  the  preference  of  commercial  standards  over  military-unique  standards.  Most 
programs  require  multiple  specification  types;  too  many  NDI-based  programs  skip  the  step  of  gener¬ 
ating  a  system  specification  and  proceed  directly  to  the  NDI  procurement  specification.  This  will 
result  in  a  lack  of  documentation  of  system-level  support  requirements  that  need  to  be  integrated  and 
maintained  with  the  system.  Table  1  identifies  some  of  the  significant  differences  between  these  three 
types  of  specifications. 

Notice  that  the  NDI  specification  requires  a  substantial  amount  of  interpretation  of  operational 
requirements  in  order  to  generate  the  product-oriented  technical  requirements.  This  is  the  primary 
contribution  of  the  system  partitioning  effort  to  the  generation  of  NDI  specifications.  Many  of  the 
decisions  made  during  the  system  partitioning  for  an  NDI  acquisition  would  be  deferred  in  a  devel¬ 
opment  acquisition  to  a  later  design  phase.  In  the  NDI  acquisition,  coordination  of  the  draft  specifi¬ 
cation  with  industry  can  serve  many  of  the  same  functions  as  the  design  phases  in  a  development 
with  regard  to  defining  specifications.  Several  draft  reviews  may  be  required  for  complex  systems. 

In  addition  to  these  characteristics,  specifications  for  NDI  must  be  sensitive  to  the  type  or  types  of 
NDI  to  be  supported  by  the  specification.  See  table  2. 
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Table  1 .  Differences  between  specification  types. 


System  Specification 

NDI  Specification 

Development  Specification 

Operational  requirements 
oriented 

Product  requirements  oriented 
(i.e.,  procurement  oriented) 

Technical  requirements  oriented  (i.e., 
design  oriented) 

Technical  requirement 
goals  specified  within 
acceptable  ranges 

Minimum  requirements  for 
acceptance  specified 

Minimum  requirements  for 
acceptance  specified 

“Minimum  acceptable” 
means  “what  is  needed  to 
meet  operational  goals.” 

“Minimum  acceptable”  means  ‘Ihe 
lowest  value  that  will  not  be 
rejected”  and  includes  technical 
risk  allowances  to  ensure  meeting 
operational  goals. 

“Minimum  acceptable”  means  “the 
lowest  value  that  will  not  be  rejected” 
and  includes  technical  risk  allowances 
to  ensure  meeting  operational  goals. 

Technical  parameters 
subject  to  tradeoff 

Technical  parameters  strictly 
required 

Technical  parameters  subject  to 
tradeoff  within  weli-defined  criteria 

Performance  requirements 
are  stated  in  operational 
terms. 

Performance  requirements  are 
stated  in  technical  terms. 

Performance  requirements  are  stated 
in  technical  terms  plus  operational 
tradeoff  criteria. 

Performance  equals 
mission  capability. 

Performance  stated  in  “form,  fit, 
and  function”  terms 

Performance  stated  in  “form,  fit,  and 
function”  terms  plus  some  “how  to” 
terms  for  implementing  advanced 
development  features 

Quality  requirements  are  in 
Test  and  Evaluation  Master 
Plan  (TEMP). 

Quality  requirements  are  process 
oriented  with  minimum  (but 
rigorous)  inspection  and  test. 

Quality  requirements  are  product 
oriented  with  emphasis  on  inspection 
and  test. 

Human  Factors  and  Safety 
constraints  identified 

Human  Factors  and  Safety 
requirements  identified  generally 
(i.e.,  “good  enough”).  Some 
system  requirements  may  be 
addressed  in  system  integration 
and  separately  from  the  NDI 
specification. 

Human  Factors  and  Safety 
requirements  identified  in  detail 

Support  requirements  and 
constraints  are  explicitiy 
identified  and  direct 
support  plans. 

Support  requirements  are  only  in 
SOW — not  in  specification. 

Support  constraints  are  specified; 
support  requirements  are  in 

Integrated  Logistics  Support  Plans 
and  SOW. 
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Table  2.  NDI  specification  approaches. 


NDI  Type 

Specification  Approach 

COTS,  ROTS,  etc. 

Sufficient  detail  for  purchasing  (Commercial  Item  Description  [CID]). 

Addihonal  system  requirements  may  be  documented  in  the  system 
specifications  or  in  separate  interface  specifications. 

MILOTS,  FME 

Use  existing  government  (performance)  specifications,  but  tailor  quality 
conformance  provisions  for  current  acquisition.  Separate  application-specific 
interface  specifications  and  system  sustainability  specifications  mav  be 
needed. 

MOTS 

Two  specifications:  (1 )  purchase  description  for  unmodified  item  and  (2) 
process  and/or  interface  specification  for  executing  the  modification.  Added 
information  for  special  quality  requirements  may  be  in  SOW. 

Integration  NDI 

System  specification  tailored  to  meet  NDI  acquisition  criteria.  May  be 
supported  by  additional  purchase  descriptions  for  off-the-shelf  items  and 
process/interface  specifications  for  any  modifications. 

Off-the-shelf  Full 

System  (system  is  not 
pre-partitioned) 

Operational  requirements  are  restated  as  User  Specifications,  including  full 
functional  descriptions,  environmental  profiles,  and  interface  specifications. 
Suppliers  are  allowed  to  generate  technical  approaches  in  response. 

(Acquirer  must  include  extensive  analysis  of  technical  approach  in  source 
selection.) 

The  most  critical  problem  in  developing  good  NDI  specifications  is  resolving  unknown  and  vague 
requirements  prior  to  using  the  specification  for  procurement.  In  a  developmental  acquisition,  most 
of  Ae  resolution  of  these  issues  is  through  engineering  analysis,  prototyping,  and  testing.  These  acti¬ 
vities  may  be  required  for  NDI  acquisitions,  but  NDI  has  more  effective  means  available  because  the 
products  already  exist.  These  means  include 

•  market  analysis, 

•  supplier  commenting,  and 

•  screening  tests. 

The  market  analysis  fills  in  many  of  the  requirements  that  would  normally  be  deferred  until  the  sys¬ 
tem  design  had  progressed  to  a  sufficiently  detailed  level  to  expose  the  interface  issues.  These  types 
of  requirements  tend  to  be  TBDs  in  most  system  specifications  because  the  detailed  design  work  is 
performed  by  the  developer  rather  than  the  system  specifier.  The  market  analysis  will  also  reveal 
information  helpful  in  performing  the  system  partitioning,  so  the  effort  actually  is  less  than  the  nor¬ 
mal  engineering  analysis  efforts  needed  to  resolve  the  same  details  in  a  developmental  effort.  As 
draft  specifications  become  available,  they  can  be  issued  for  industry  comment.  Industry  comment¬ 
ing  accomplishes  several  things  at  once: 

1 .  known  specifications  can  be  commented  upon  for  realism,  doability,  and  potential  cost  impact, 

2.  the  requirements  can  be  verified  for  comprehensibility,  completeness,  and  clarity,  and 

3.  realistic  targets  can  be  identified  for  the  TBD  specifications. 

Issuing  the  specifications  for  comment  should  be  done  through  the  contracting  office,  especially  for 
the  non-off-the-shelf  items.  This  way,  the  contract  files  will  have  the  information  to  satisfy  the  legal 
requirements  for  a  fixed-price  contract.  When  possible,  screening  tests  can  be  conducted  as  either  a 
part  of  the  market  analysis  or  as  part  of  the  proposal  process  rather  than  as  part  of  acceptance  testing. 
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The  tests  can  be  structured  to  identify  any  missing  interface  and  support  requirements.  In  addition  to 
resolving  the  TBDs,  the  vague  requirements,  and  other  unknowns,  the  resulting  specifications 
become  well  defined  and  well  understood  by  the  prospective  suppliers.  These  conditions  reduce 
costs  and  risks  and  increase  the  resultant  system  quality. 

The  various  government  specifications  (Military  Specifications,  Federal  Specifications,  Military 
Standards,  Federal  Standards,  and  other  agency  standards)  should  be  consulted  but  used  with  cau¬ 
tion.  For  instance,  the  MIL-SPEC  system  of  specifications,  standards,  and  handbooks  constitute  the 
coiporate  memory  of  the  DoD.  These  documents  contain  requirements,  conditional  requirements, 
desires,  and  good  advice,  but  a  requirement  for  one  program  may  be  only  good  advice  for  another. 
The  conditional  requirements  are  particularly  troubling  for  both  the  acquisition  agent  and  the  sup¬ 
plier  because  they  involve  tradeoffs  between  performance  issues,  cost  issues,  risk  issues,  and  sched¬ 
ule  issues.  These  tradeoffs  justify  involving  extensive  technical  and  programmatic  expertise  in  the 
specification  writing  process  that  can  be  carried  into  the  dialog  with  industry  in  preparation  for  an 
acquisition.  It  is  important  to  include  expertise  in  a  system  design  team  highly  conversant  in  the 
relevant  government  documents  to  sort  out  the  issues  raised.  This  is  another  of  the  roles  of  the 
“smart  buyer”  function  played  by  a  government  laboratory.  Current  DoD  policy  is  to  cite  industry 
standards  rather  than  military  specifications,  and  to  include  military  requirements  explicitly  rather 
than  by  reference.  However,  this  policy  only  addresses  a  small  piece  of  the  specification  writing 
problem.  The  various  government  requirements  documents  do  contain  pertinent  requirements,  but 
these  requirements  usually  require  interpretation  (tailoring)  for  use  with  NDI.  The  goal  is  to  provide 
specifications  in  language  easily  interpreted  by  the  suppliers,  consistent  with  supplier/market  prac¬ 
tices,  and  granting  suppliers  the  maximum  latitude  to  meet  the  acquisition  requirements  with  high- 
quality  products. 

All  requirements  originate  with  the  system  being  designed.  However,  the  system  design  intention¬ 
ally  adopts  as  many  standards  from  pre-existing  sources  as  possible.  This  promotes  a  long  system 
design  life,  good  system  supportability,  cost-effective  system  acquisition  and  support,  the  ability  to 
use  NDI,  and  multiple  sources  of  system  items  and  support.  Procurement  specifications  for  NDI 
should  include  all  of  the  requirements  essential  to  incorporate  the  items  into  the  system  design  and 
nothing  else.  (Ideally,  a  simplified  specification  form  called  a  Commercial  Item  Description  [CID] 
can  be  used  for  most  COTS.)  There  may  be  other  system  requirements,  but  these  should  be  in  other 
specifications.  For  any  given  item  in  the  system  design,  there  will  be  requirements  unique  to  the  mil¬ 
itary  application  of  the  system  and  requirements  that  are  related  to  the  technology  being  implemented 
in  the  system  design.  The  military-unique  requirements  will  be  drawn  from  various  military  stan¬ 
dards,  such  as  MIL-STD-461  for  electromagnetic  compatibility  issues  (the  military  has  many  EMC 
issues  that  far  exceed  commercial  EMC  standards).  However,  military-unique  requirements  are  best 
incorporated  explicitly,  where  possible,  and  tailored  to  the  specific  system  application.  Incorporate 
requirements  by  reference  only  as  a  last  resort,  when  the  volume  of  information  associated  with  the 
requirement  is  extensive  even  after  tailoring  the  requirement.  Commercial  standards  may  be  cited  by 
reference,  but  they  should  still  be  tailored  to  the  system  application.  System  proprietary  standards 
are  all  requirements  not  supported  by  an  external  reference  standard.  These  requirements  must  be 
stated  explicitly.  Where  the  system  proprietary  requirements  have  been  derived  from  engineering 
studies,  it  is  a  good  idea  to  provide  a  reference  to  the  source  material.  Use  of  an  open  architecture 
helps  to  separate  the  system  specification  issues  from  the  NDI  product  procurement  specification 
issues  so  that  these  specifications  can  each  stand  on  their  own. 
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A  host  of  commercial  and  governmental  standards  are  available  from  which  to  choose.  The  selec¬ 
tion  should  be  driven  by  the  system  partitioning  decisions,  but,  all  other  things  being  equal,  the  order 
of  preference  should  be  as  follows: 

•  DoD-adopted  industry  standards. 

•  International  industry  standards  (ISO). 

•  National  industry  standards  (ANSI). 

•  Industry-specific  standards  (IEEE,  SAE,  etc.). 

•  Federal  standardization  standards  (FIPS,  FED-STD). 

•  Federal  interface  and  performance  specifications  (FED-SPEC). 

•  Agency-specific  interface  standards  (MIL-STD). 

•  Agency-specific  performance  standards  (MIL-SPEC). 

•  Agency-specific  guidance  (MIL-HDBK). 

•  System-specific  specifications  for  existing  MILOTS. 

•  System-specific  specifications  for  existing  FME. 

•  Market-driven  specifications. 

•  Commercial  proprietary  specifications. 

All  other  things  usually  are  not  equal.  Also,  the  order  of  preference  of  selecting  standards  is  not  the 
same  as  the  order  of  preference  in  considering  NDI  candidates  (see  SYSTEM  PARTITIONING 
FOR  NDI). 

In  1989,  this  Center  conducted  an  experiment  in  specifying  NDI.  The  system  requirement  was  for 
some  specialized  electronics  gear  to  be  used  in  the  Persian  Gulf  Theater.  Several  different  types  of 
NDI  existed  to  fulfill  the  system  needs.  The  options  included  a  full  system  (very  expensive)  requir- 
ing  minor  modifications  that  was  available  from  a  single  source,  COTS  requiring  some  integration 
and  many  minor  modifications  from  many  commercial  sources,  ROTS  requiring  some  integration 
but  no  modifications,  and  some  MILOTS  requiring  minor  modifications.  Altogether,  there  were  sev¬ 
eral  dozen  potential  sources  of  supply.  About  half  of  these  sources  were  strictly  commercial,  while 
the  other  half  were  companies  routinely  doing  business  with  the  military.  The  Center  prepared  two 
specification  packages — one  for  the  commercial  market  and  one  for  the  military  market.  The  com¬ 
mercial  market  specification  contained  no  references  to  external  specifications  or  standards;  all  of  the 
requirements  were  heavily  tailored  and  highly  explicit.  The  military  market  specification  referenced 
many  military  and  industry  specifications,  but  each  reference  was  heavily  tailored  to  ensure  that  the 
actual  requirements  were  consistent  between  both  packages.  The  packages  were  broadcast  in  sepa¬ 
rate  announcements  to  result  in  separate  proposals  with  the  caveat  that  either  procurement  might  be 
canceled.  We  requested  specification  comments  and  very  preliminary  cost  proposals  for  planning 
purposes.  After  receiving  initial  proposals,  we  conducted  a  bidder’s  conference.  There  were  actu¬ 
ally  separate  doors  representing  each  announcement,  but  they  led  to  a  common  room.  Commercial 
responders  to  the  commercial  market  specification  provided  cost  estimates  averaging  25  percent 
below  the  military  market  responders  to  the  commercial  specification.  Military  market  responders  to 
the  military  market  specification  provided  cost  estimates  averaging  about  25  percent  below  the  com¬ 
mercial  sources  responding  to  the  military  market  specification.  The  technical  responses  and  cost 
estimates  were  consistent  between  the  two  lower  tier  and  the  two  upper  tier  responses  that  resulted. 
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In  other  words,  the  value  of  the  technology  to  meet  the  requirements  was  the  same  in  each  market, 
but  there  was  also  a  cost  premium  for  the  perceived  risk  of  providing  a  product  to  a  different  market 
with  which  the  company  was  unfamiliar.  This  was  particularly  noticeable  with  companies  respond¬ 
ing  to  both  announcements.  While  the  experiment  constitutes  a  datum  of  one,  it  illustrates  the  impor¬ 
tance  of  communicating  requirements  to  a  market  in  language  tailored  to  that  market. 

Finally,  specifications  must  sometimes  incorporate  some  special  enabling  features  for  NDI  support 
that  would  not  usually  be  considered.  The  most  common  of  these  requirements  is  for  various  fea¬ 
tures  to  support  warranty  enforcement.  One  common  feature  is  a  time-totalizing  meter  to  allow  war¬ 
ranty  time  to  be  measured  on  usage  hours  rather  than  on  calendar  time.  Another  feature  might  be  a 
lockout  to  prevent  tampering  inside  a  warranted  item.  Special  packaging  might  be  required  to  adapt 
the  item  to  a  new  environment;  however,  the  “what”  elements  are  the  requirements  rather  than  the 
“how  to”  issues.  The  system  designer  must  be  aware  of  these  factors  and  features  to  include 
appropriate  requirements  in  the  specifications. 

Knowledge  of  the  NDI  issues,  awareness  of  the  markets,  incorporating  appropriate  technical 
expertise  in  standing  requirements  and  in  the  technologies,  and  diligence  in  attention  to  detail  all 
contribute  to  good  specifications  for  NDI  acquisitions. 

STATEMENT  OF  WORK  (SOW)  REQUIREMENTS 

A  statement  of  work  describes  the  tasks  to  be  performed  by  a  supplier  in  the  course  of  performing 
a  contract.  These  tasks  should  be  constructed  to  embody  those  requirements  that  are  not  covered  in 
the  specifications  or  in  delivered  data.  The  typical  requirements  associated  with  NDI  deal  with  con¬ 
ducting  additional  analyses  or  tests  to  generate  quality,  reliability,  or  support  information  that  is  not 
needed  by  the  normal  product  market.  Other  tasks  may  involve  modifications  to  the  product  or 
delivered  data  to  adjust  for  differences  between  the  market  for  which  the  product  was  designed  ver¬ 
sus  the  application  to  which  it  is  being  adapted.  Of  course,  additional  requirements  arise  from  the 
management  of  the  contract,  such  as  design  reviews  for  Integration  NDI.  Whatever  these  tasks  may 
be,  the  differences  between  the  normal  business  practices  of  NDI  suppliers  and  the  normal  practices 
of  DoD  contracting  can  be  substantial.  It  is  important  to  scope  and  phrase  the  task  requirements  such 
that  adequate  best  industry  practices  are  allowed  while  DoD  contracting  requirements  are  met.  Devi¬ 
ations  from  standard  practices  will  cost  extra  and  can  defeat  some  of  the  immediate  schedule  and 
cost  benefits  of  NDI. 

Wherever  possible,  use  purchasing  techniques  and  conduct  modifications  to  NDI  “in  house”  to  the 
acquisition.  Modifications  may  often  be  done  on  the  final  product  or  data  after  it  has  been  accepted 
from  the  original  supplier.  The  limitations  to  this  approach  arise  when  the  modification  would  inval¬ 
idate  a  warranty  or  when  the  modification  is  so  intrinsically  associated  with  the  product  manufacture 
that  it  must  be  done  on  the  production  line.  However,  modifications  done  after  acceptance  allow 
greater  control  and  flexibility  for  future  modifications  that  can  be  critical  to  applications  with 
dynamic  operational  requirements. 

Modifications  to  NDI  must  often  be  done  during  the  production  process.  For  example,  consider 
the  mundane  problem  of  having  the  NDI  painted  the  right  color.  Ideally,  one  would  accept  NDI  with 
its  original  paint;  however,  the  original  paint  may  be  international  orange  where  a  camouflage  color 
scheme  might  be  required.  This  circumstance  justifies  changing  the  manufacturer’s  processes  if  the 
supplier  is  willing  to  do  so  (many  commercial  suppliers  will  not  change  their  production  line  unless 
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there  are  very  substantial  minimum  orders  greatly  in  excess  of  a  typical  acquisition  program’s 
needs).  In  these  cases,  appropriate  tasks  should  be  included  in  the  SOW  to  ensure  that  the  produc¬ 
tion  process  flow  and  quality  levels  are  maintained. 

Sometimes,  additional  tasks  may  be  needed  to  provide  added  data  for  the  acquisition  agency’s  use 
or  to  condition  the  product  for  special  application  environments.  The  Navy  has  a  standard  policy  of 
applying  Environmental  Stress  Screens  to  NDI  hardware  and  Application  Stress  Screens  to  NDS. 
Each  acquisition  must  be  assessed  to  determine  if  these  screens  should  be  done  by  the  supplier  or 
after  delivery.  If  they  are  to  be  done  by  the  supplier,  the  SOW  must  contain  the  tasking.  If  they  are 
to  be  done  by  a  test  agent  other  than  the  supplier,  SOW  tasks  my  be  required  to  keep  warranties  in 
effect  or  to  fix  material  found  to  be  discrepant. 

Also,  special  problems  with  data  acquisition  may  require  added  tasks  in  the  SOW  See  DATA 
ACQUISITION. 

In  any  case,  SOW  requirements  should  be  kept  to  a  minimum  and  should  be  sensitive  to  the  mar- 
ketplace(s)  of  the  viable  sources  of  supply.  Industry  reviews  of  draft  SOWs  are  very  useful  in  avoid¬ 
ing  cost  driver  requirements  and  in  identifying  issues  that  should  be  included  or  deleted.  The  indus¬ 
try  review  can  also  ensure  that  the  SOW  requirements  are  well  defined  and  clearly  stated  for  use  by 
the  suppliers. 


DATA  ACQUISITION 

Data  is  information  in  a  specific  format.  Although  the  dictionary  may  not  agree,  this  definition 
works  well  in  DoD  contracting.  The  acquisition  project  needs  information,  but  data  is  what  is  actu¬ 
ally  obtained.  All  data  must  be  acquired  in  accordance  with  a  Contract  Data  Requirements  List 
(CDRL).  The  CDRL  establishes  both  administrative  requirements  (delivery  date,  receiving  activity, 
number  of  copies)  and  technical  requirements  (data  content,  preparation  instructions,  formats).  The 
technical  requirements  are  detailed  in  Data  Item  Descriptions  (DID)  referenced  and  tailored  in  the 
CDRL.  This  applies  to  all  DoD  acquisitions. 

In  NDI  acquisitions,  most  (but  not  all)  of  the  information  will  already  exist;  however,  many  of  the 
information  items  may  be  proprietary  or  in  a  form  that  is  not  suitable  for  the  current  acquisition. 

How  the  data  is  acquired  can  severely  affect  the  contract  price.  Inappropriate  data  acquisition  mea¬ 
sures  in  NDI  acquisitions  have  resulted  in  data  packages  costing  as  much  as  90  percent  of  the  full 
acquisition  price.  A  determination  must  be  made  as  to  the  most  cost-effective  means  of  obtaining 
information  that  does  not  currently  exist.  Usually,  some  form  of  tailored  screening  test  will  become 
the  most  inexpensive  approach.  The  general  rules  in  acquiring  data  all  apply,  but  some  special  con¬ 
siderations  for  NDI  acquisitions  also  apply. 

In  general,  buy  only  the  minimum  essential  data  in  its  naturally  occurring  formats.  In  buying  NDI, 
the  formats  will  be  predetermined  for  virtually  all  data.  This  is  in  contrast  to  developmental  acquisi¬ 
tions,  where  much  data  is  being  developed  as  the  product  is  being  developed  and  where  the  desired 
format  can  be  used  as  the  data  is  developed.  In  developmental  acquisitions,  the  format  required  for 
end  use  of  the  information  is  commonly  specified  for  the  data;  this  can  have  expensive  consequences 
in  NDI  acquisitions.  Furthermore,  COTS  acquisitions  may  encounter  proprietary  rights  in  data  that 
may  make  it  impractical  to  acquire  certain  data  elements.  However,  just  because  information  may  be 
proprietary  does  not  mean  that  it  cannot  be  acquired.  The  desired  information  may  be  proprietary  in 
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one  form  and  not  in  another  because  of  its  association  with  other  information  in  certain  data  formats. 
Also,  certain  data  proprietary  items  may  be  available  at  a  reasonable  cost  but  with  disclosure  restric¬ 
tions.  As  a  result,  data  format  can  be  a  primary  driver  of  data  costs.  For  NDI  acquisitions,  data  for¬ 
mats  have  already  been  set  by  the  market,  so  deviations  from  these  formats  will  always  increase 
costs  without  changing  the  content  of  the  information  that  is  desired.  Following  these  general  data 
acquisition  steps  will  help  to  ensure  that  all  of  the  information  is  obtained  in  the  most  cost-effective 
manner: 

1.  Establish  the  information  requirements  of  the  acquisition. 

2.  Determine  the  end  uses  of  the  information,  noting  where  one  information  element  may  be  used 
for  several  end  uses.  (It  is  useful  to  establish  a  matrix  of  information  elements  and  end-use 
elements  in  order  to  quickly  visualize  these  relationships.) 

3.  Determine  which  information  elements  already  exist  and  note  the  data  form  for  each  element. 

4.  Specify  the  end-use  format  requirements,  maintaining  as  much  flexibility  in  format  as  possible. 

5.  Determine  modifications  to  existing  data  that  may  be  required  to  make  the  information  usable 
in  the  required  end  use. 

6.  Establish  tasks  for  developing  data  for  information  elements  that  do  not  currently  exist. 

7.  Establish  approaches  for  obtaining  existing  data. 

8.  Draft  the  CDRL  package,  obtaining  industry  comment  when  appropriate. 

Given  the  importance  of  data  to  the  life-cycle  support  of  a  system,  these  steps  should  receive  as 
much  attention,  care,  and  expertise  as  the  generation  of  specifications. 

Accurately  dctcmiiniri'j  the  information  requirements  is  usually  the  most  challenging  of  all  these  steps 
for  any  acquisition  Tlie  following  information  elements  are  not  meant  to  be  all  inclusive;  however,  this 
list  may  prove  helpf  ul  to  identify  data  that  might  otherwise  be  overlooked.  Please  note  that  most  of  these 
information  elementN  are  a\  ailable  within  the  government  rather  than  from  the  supplier. 

User  Requiremeius  /nfonnalion 

Operational  Requirements 
Operational  Doctrine 
Operational  .Support  Concepts 
User  SkilN 
Maintainer  Skills 
User  Interface  Standards 

Performance  Information 

Application  Performance  Conformance 
Quality  Data 
Inspection  Data 
Field  Data 
Test  Data 

Environmental  Conformance 
Analysis  Data 
Field  Data 
Test  Data 
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Electromagnetic  Compatibility  Conformance 
Field  Data 
Test  Data 

Safety  Certifications 

Analysis  Data 
Test  Data 

Supportability  Information 

Integrated  Logistics  Support  Plan  (ILSP)  [and  subsidiary  support  plans] 

Failure  Data 
Maintenance  Data 
Warranty  Data 
Repair  Item  Cost  Data 
Technical  Manuals 
Operator  Manuals 
Training  Data 

Facilities  Management  Data 

Installation  Information 

Interface  Control  Data 
Installation  Control  Data 
Installation  and  Checkout  Procedures 

Life-Cycle  Control  Information 

Interface  Standards  (at  each  level  of  complexity) 

Design  Control  Data 
Production  Control  Data 
Modification  and  Upgrade  Control  Data 
Configuration  Management  Data 

Project  Management  Information 

Design  Review  Information 
Project  Metrics 

Requirements 

Quality/Defects 

Reliability 

Technical  Performance  and  Design  Characterization 
Productivity 

Project  Performance  (cost  and  schedule) 

Project  Administrative  Information 
Progress 
Financial 

Work  Breakdown  and  Tasking  Information 

Most  of  the  information  needed  for  a  project  is  not  obtained  from  the  product  supplier.  Most  of 
the  data  requirements  are  for  use  internal  to  the  government  (and  mostly  within  the  requiring 
agency),  and  most  of  the  information  elements  are  resident  with  the  government,  being  the  result  of 
the  requirements  definition  process  or  the  decisions  needed  to  assemble  a  procurement  package. 
Therefore,  the  most  cost-effective  means  of  obtaining  the  information  needed  from  the  supplier  is  to 
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get  existing  data  in  its  native  formats  and  to  extract  or  convert  the  information  to  the  required  format. 
If  the  information  is  available  in  digital  form  that  can  be  transferred  through  a  recognized  standard, 
the  conversion  from  supplier  to  government  formats  is  even  more  efficient. 

To  promote  the  efficient  sharing  of  information  between  industry  and  governmental  entities,  DoD 
has  established  the  CALS  initiatives.  (CALS  currently  stands  for  Concurrent  Acquisition  and  Life- 
Cycle  Support.  The  definition  of  the  CALS  acronym  has  been  changed  several  times,  but  the 
acronym  has  remained  stable.)  The  CALS  initiatives  have  adopted  several  formats  for  exchanging 
data.  These  include  the  Initial  Graphics  Exchange  Specification  (IGES)  (a  vector  format  for  drawing 
information),  the  Joint  Engineering  Documentation  Management  Information  Control  System  (JED- 
MICS)  specification  (a  raster  image  format  that  is  effectively  a  Tagged  Image  File  Format  (TIFF) 
with  some  added  file  header  information),  the  Standard  Graphics  Markup  Language  (SGML)  (a  text 
plus  graphics  format  that  is  a  superset  of  the  Hypertext  Markup  Language  (HTML)  used  to  create 
Internet  web  pages  and  of  the  Help  Metafile  Language  used  in  many  computer  “help”  and  “tutorial” 
applications.),  and  the  Computer  Graphics  Metafile  (CGM)  format  (a  vector  and  raster  image  format 
for  computer  presentation  files).  The  CALS  standards  are  widely  accepted  and  used  by  industry,  and 
commercial  tools  are  available  to  translate  between  the  various  formats  and  to  these  formats  from  the 
various  proprietary  file  formats  used  by  the  various  computer  tools  that  companies  use  to  create  and 
manage  their  products.  Most  companies  doing  business  with  DoD  now  support  CALS;  however, 
many  of  the  small  commercial  businesses  supplying  COTS  items  do  not  have  CALS  tools.  Where 
CALS  tools  do  not  exist,  but  the  data  is  in  digital  form,  almost  any  mutually  agreeable  format  can  be 
made  to  work.  The  most  common  include  the  popular  printer/plotter  languages  (encapsulated  post¬ 
script  and  the  Hewlett  Packard  standard  languages),  for  which  tools  exist  to  convert  from  these  for¬ 
mats  to  CALS  formats. 

When  preparing  to  receive  data  in  digital  form,  it  is  an  excellent  idea  to  conduct  test  transfers  of 
simple  test  patterns  that  contain  one  each  of  all  of  the  digital  entities  that  will  make  up  the  data  files 
to  be  delivered.  This  is  because  the  digital  data  is  usually  being  translated  out  of  a  proprietary  format 
into  the  specified  export  format.  The  implementation  of  the  data  translators  may  be  incomplete. 

Also,  there  can  be  various  “flavors”  of  the  commercial  standards  that  might  be  involved  that  will 
have  subtle  and  incompatible  variations.  Some  of  these  incompatibilities  will  cause  some  of  the  digi¬ 
tal  data  to  be  lost  or  corrupted.  The  test  transfers  can  discover  any  problems  that  might  exist  and 
allow  workarounds  and  fixes  to  be  identified.  Sometimes  the  information  will  be  slightly  refor¬ 
matted  (text  fonts  might  be  changed,  for  instance);  other  times  the  information  will  be  unreadable. 
Generally,  as  long  as  the  information  is  readable  and  intact,  it  will  be  acceptable. 

A  much  more  serious  problem  arises  when  the  supplier  information  is  created  for  a  market  that  is 
fundamentally  incompatible  with  the  intended  application.  The  most  common  instance  occurs  when 
the  commercial  item  is  intended  for  specially  trained  and  highly  skilled  technicians,  but  the  intended 
application  is  being  supported  by  junior  field  technicians.  In  these  circumstances,  the  commercial 
technical  manuals  are  likely  to  be  written  at  several  reading  grade  levels  above  the  service  standard. 

If  more  than  10  percent  of  a  commercial  data  item  must  be  rewritten  or  converted  to  be  compatible 
with  the  intended  application,  then  the  entire  item  should  be  rewritten.  If  only  minor  modifications 
are  required,  a  data  supplement  can  be  generated  to  incorporate  the  needed  changes;  however,  it  is 
important  to  ensure  that  the  information  flow  and  continuity  of  the  data  item  is  not  disrupted  when  a 
supplement  is  employed.  This  may  mean  that  added  information  might  be  included  in  the  data  sup¬ 
plement  to  retain  the  information  continuity.  Rewriting  technical  manuals  can  be  very  expensive,  but 
the  operators  and  maintainers  must  be  viewed  as  part  of  the  overall  system.  The  most  effective 
means  of  communicating  system  technical  information  to  operators  and  maintainers  should  be 
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considered  in  the  system  planning  and  integration  and  should  consider  total  life-cycle  costs.  Since 
personnel  are  high-tumover  elements  of  the  system,  the  costs  of  communicating  technical  informa¬ 
tion  to  them  through  technical  manuals  and  training  become  very  significant.  Technical  manual  costs 
are  relatively  fixed  while  training  costs  are  recurring  costs,  so  life-cycle  cost  decisions  tend  to  favor 
good  technical  manuals. 

Many  types  of  information  need  to  be  incorporated  into  system  planning.  Much  of  this  informa¬ 
tion  is  found  in  different  forms  for  NDI  in  contrast  to  developmental  acquisitions.  The  data  for 
COTS  Items  is  formatted  for  commercial  uses  by  users  of  variable  sophistication.  Developmental 
acquisition  data  is  usually  created  in  the  format  in  which  it  will  be  used.  Much  of  the  system  support 
information  must  be  obtained  from  suppliers  of  NDI  but  reformatted  for  use  in  the  DoD  support  sys¬ 
tem.  In  addition,  the  quality  of  the  information  from  the  various  NDI  sources  may  be  questionable. 
The  confidence  levels  associated  with  quality  and  reliability  information  from  NDI  sources  may  be 
different  from  that  needed  for  a  particular  system  application.  These  differences  must  be  resolved  in 
order  to  use  the  information  appropriately.  Tests  are  often  required  in  order  to  develop  sufficiently 
high  confidence  le\  els  and  to  resolve  information  quality  issues.  The  tests  are  usually  incorporated 
into  the  tailored  screens,  but  if  they  are  not,  they  should  be  budgeted  as  part  of  the  data  acquisition 
effort. 

When  acquiring  F.ME  or  FCOTS,  be  sure  to  plan  for  the  time  and  expense  of  translating  the 
information  into  English,  if  required.  Also,  ensure  that  specifications  and  other  support  information 
are  available  in  EngliNh  as  well  as  the  native  language.  Make  sure  that  translation  costs  are  identified 
for  the  procurements  and  also  for  the  maintenance  of  technical  documentation. 

What  is  the  most  cost-effective  means  of  obtaining  high-quality  information  to  support  good  sys¬ 
tem  decisions'?  There  are  a  variety  of  ways  to  obtain  this  information  for  NDI.  For  instance,  to  sup¬ 
port  a  system  deplo>  ed  in  a  ship  in  the  Indian  Ocean  is  a  substantially  different  problem  from  sup¬ 
porting  the  same  s\  stem  in  San  Diego,  California.  To  achieve  high  levels  of  operational  availability, 
the  support  planning  must  have  good  information  on  the  product  reliability  and  failure  rates.  One  can 
conduct  a  reliabilit>  anal>  sis  of  the  design,  conduct  an  analysis  of  field  failure  data,  review  warranty 
data,  review  supplier  test  data,  conduct  reliability  qualification  tests,  or  perform  some  combination  of 
these  methods.  GeneralK.  field  data  and  warranty  experience  provide  the  highest  confidence  data, 
but  the  confidence  in  this  data  can  be  eroded  by  differences  in  application  environments,  lack  of 
product  maturits.  and  undisclosed  methods  of  converting  raw  information  into  published  data.  If  a 
COTS  item  was  designed  for  use  in  home  environments  in  San  Diego,  there  is  very  low  confidence 
in  the  field  failure  data  for  its  use  in  support  planning  for  a  deployed  shipboard  application; 
nevertheless,  the  item  ma\  be  perfectly  acceptable  for  this  application.  An  analysis  of  the  design 
trught  give  a  qualitatixe  confidence  that  it  is  sufficiently  robust  for  use  aboard  ship,  but  the  quantita¬ 
tive  confidence  levels  in  the  information  derived  from  such  an  analysis  may  require  extra  pipeline 
spares  of  an  expensi\  e  end  item  in  order  to  assure  good  operational  availability.  Testing  may  provide 
high-confidence  information,  but  the  test  itself  may  be  prohibitively  expensive  for  a  highly  reliable 
item.  This  is  especially  true  if  the  system  is  being  acquired  for  only  a  few  dozen  installations.  Also, 
supplier  tests  may  be  inadequately  documented  to  determine  their  accuracy  or  applicability.  (Some  ’ 
types  of  tests  can  be  grossly  affected  by  the  test  setup  or  test  procedures  so  that  reported  performance 
is  much  better  than  what  can  be  realized  in  the  field.  This  is  especially  true  of  environmental  tests 
such  as  impact  shock  and  vibration.)  The  normal  solution  requires  an  integration  of  all  of  the  avail¬ 
able  information  together  with  screening  tests.  The  screening  tests  are  constructed  to  eliminate  fac¬ 
tors  that  degrade  the  confidence  associated  with  other  sources  of  information.  This  allows  the  tests 
to  be  constructed  to  be  affordable  but  still  allows  decisions  to  be  based  on  the  best  available 
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information.  The  goal  is  to  use  the  available  information  to  the  maximum  extent  possible,  supple¬ 
menting  it  with  newly  developed  information  as  needed  to  achieve  confidence  levels  that  are  good 
enough.  A  good  indication  of  “good  enough”  is  when  information  from  the  different  sources  all 
agree  within  the  acceptable  uncertainties. 

In  general,  at  least  as  much  effort  must  be  put  into  acquiring  good  information  as  in  acquiring  the 
products.  This  is  especially  true  of  NDI  acquisition.  Bringing  the  appropriate  expertise  to  bear  in 
developing  the  information/data  requirements  and  tailoring  the  information  development  activities  is 
at  least  as  important  as  applying  good  expertise  to  the  system  design  and  development  of  specifica¬ 
tions.  This  problem  is  easier  for  NDI  in  the  respect  that  so  much  more  information  can  generally  be 
made  available.  The  problem  is  more  difficult  in  that  the  data  must  be  interpreted  from  its  market 
perspective  into  the  system  application  perspective.  It  is  a  good  idea  to  submit  draft  versions  of  the 
CDRL  to  industry  for  comment  to  ensure  that  no  unnecessary  cost  driver  requirements  are  included. 
The  CDRL  and  the  referenced  DIDs  must  be  tailored  carefully  to  obtain  all  of  the  information 
needed  in  acceptable  and  usable  forms  while  allowing  for  the  high  degree  of  variability  that  may 
exist  in  the  NDI  market. 


SOLICITATION  AND  SOURCE  SELECTION  TIPS 

Source  selection  can  be  based  on  either  least  overall  cost  or  best-value  criteria.  It  is  essential  to 
determine  which  set  of  criteria  is  to  be  used  and  to  develop  appropriate  supporting  solicitation  docu¬ 
mentation. 

When  least  overall  cost  criteria  are  used,  the  cost  to  the  government  is  the  primary  determining 
factor  in  source  selection,  excluding  candidates  that  are  technically  unacceptable.  Often,  only  the  bid 
price  is  used  to  determine  costs,  but  only  because  other  costs  are  left  undetermined  (because  the  bid 
cost  is  the  only  significant  cost,  because  the  other  costs  are  not  variable  between  bidders,  or  because 
the  acquiring  agency  has  neglected  to  develop  appropriate  tools  for  evaluating  other  costs).  Other 
cost  factors  can  be  used.  The  two  most  common  are  Design-to-Cost  (DTC)  and  Life-Cycle  Cost 
(LCC).  DTC  criteria  include  the  bid  cost  plus  costs  for  acceptance  and  post-delivery  requirements 
(such  as  installation  and  checkout  and  modifications).  For  most  NDI  acquisitions,  acceptance  and 
post-delivery  costs  are  equal  for  all  candidates,  so  the  bid  cost  is  the  only  significant  determining  fac¬ 
tor.  However,  if  NDI  modifications  are  required,  there  may  be  significant  differences  in  non-bid 
costs  that  should  be  accounted  for  in  the  source  selection.  LCC  criteria  evaluate  the  costs  for  the 
entire  life  of  the  system.  Since  there  may  be  significant  differences  in  quality  and  reliability  between 
candidates,  LCC  criteria  can  make  a  substantial  difference  in  the  source  selection  from  bid  cost. 
However,  to  employ  LCC  criteria,  a  LCC  model  must  be  developed  and  validated.  Failure  to  vali¬ 
date  the  model  prior  to  its  use  will  lead  to  protests  by  non-selected  sources,  particularly  those  who 
have  a  lower  bid  cost  than  the  selected  source.  The  validation  of  the  cost  model  involves  test 
applications  to  existing  systems  where  the  costs  are  known  and  well  documented  to  show  that  the 
model  accurately  predicts  the  actual  experience.  The  validation  process  is  both  lengthy  and  expen¬ 
sive,  so  LCC  is  seldom  applied  to  NDI  acquisitions  unless  a  validated  model  already  exists.  Another 
problem  with  LCC  arises  because  many  usable  models  are  forced  by  cost  factors  that  are  hard  to 
determine  prior  to  several  years  of  field  use.  Such  models  are  useful  for  reprocurements  and  support 
planning,  but  not  for  source  selection.  Nevertheless,  LCC  criteria  are  the  best  to  use  whenever  least 
overall  cost  is  appropriate  for  source  selection. 
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The  best  least  overall  cost  criteria  for  use  in  NDI  acquisitions  are  Differential  Life-Cycle  Costs 
(DLCC).  In  DLCC,  a  simplified  model  is  constructed  that  contains  only  those  factors  that  can  be 
affected  by  the  supplier.  These  might  include  the  bid  cost,  reliability/failure  rate  factors  (affecting 
the  cost-per-failure  support  costs),  the  reading  grade  level  and  effective  page  count  of  the  technical 
manuals  (affecting  training  costs  or  personnel  factors),  weight  and  power  and  special  service 
requirements  (affecting  installation  costs),  quality  factors  (that  affect  the  confidence  applied  to  other 
planning  numbers,  and  support  costs.  Any  life-cycle  factors  that  are  not  affected  by  elements  under 
the  supplier’s  control  are  computed  to  fixed  factors  (such  as  the  life-cycle  billet  costs  of  users  and 
mamtamers).  The  proposal  preparation  instructions  (Section  L  of  the  solicitation)  will  require  the 
proposer  to  provide  the  data  needed  to  run  the  model.  Proposers  should  be  required  to  provide 
information  to  substantiate  the  numbers  provided  for  the  DLCC.  Since  DLCC  models  are  highly 
simplified  compared  to  normal  LCC  models,  they  can  be  easily  and  quickly  evaluated.  The  need  to 
validate  the  model  can  be  eliminated  by  soliciting  industry  comments  when  the  specification,  SOW, 
and  CDRL  are  sent  out  for  comment.  Also,  the  DLCC  model  should  be  published  as  part  of  the  eval¬ 
uation  criteria  (Section  M  of  the  solicitation).  DLCC  is  the  recommended  approach  for  NDI  acquisi¬ 
tions  when  using  least  overall  cost  criteria. 

Best-value  criteria  weigh  technical  factors  more  heavily  than  cost  factors  in  the  source  selection. 
The  criteria  are  applied  in  recognition  that  paying  a  little  more  for  significant  increases  in  perfor¬ 
mance,  quality,  and  reliability  can  be  greatly  beneficial  to  the  government.  The  simplest  form  of  best 
value  merely  weighs  the  technical  evaluation  score  2,  3,  or  4  times  as  heavily  as  the  cost  evaluation 
score.  This  is  very  useful  for  research  and  development  proposals  where  there  may  be  relatively 
small  differences  between  cost  proposals  but  clear  differences  in  technical  proposals.  However,  this 
approach  does  not  accurately  evaluate  relative  cost  differences  against  technical  differences,  and  it 
invites  protests  when  there  are  large  cost  differences  between  acceptable  proposers.  It  is  not  recom¬ 
mended  for  NDI  acquisitions.  Another  best-value  approach  scores  all  proposers  against  the  lowest 
cost  acceptable  proposal.  In  this  approach,  the  source  selected  is  the  greatest  weighted  delta  in  tech¬ 
nical  score  for  the  least  weighted  delta  in  cost.  This  approach  does  work  well  for  NDI  acquisitions, 
especially  for  the  non-off-the-shelf  alternatives. 

Least  overall  cost  criteria  should  be  used  whenever  the  candidates  are  true  off-the-shelf  (no  modi¬ 
fications  or  integration  by  the  supplier),  mature  (no  NewCOTS),  and  competitive  in  the  same  market. 
Best-value  criteria  should  be  used  whenever  the  candidates  include  NewCOTS,  MILOTS  available 
from  several  suppliers  to  performance  specifications,  mixtures  between  COTS/ROTS  and  MILOTS, 
or  diverse  markets.  Much  NDI  can  be  acquired  using  purchasing  techniques  (applying  least  overall' 
cost  criteria).  In  practical  terms  for  larger  systems,  least  overall  cost  criteria  are  most  usually 
desired,  but  best- value  criteria  are  applied  because  it  is  impossible  to  determine  ahead  of  time  that  all 
candidates  will  conform  to  the  least  overall  cost  strictures.  Also,  the  high  complexity  of  large  sys¬ 
tems  tends  to  obscure  performance  factors,  so  a  best-value  approach  is  justified  to  provide  a'^more 
thorough  and  fair  treatment  of  the  performance  versus  cost  differentials. 

A  third  best-value  approach  takes  the  phrase  “most  bang  for  the  buck”  quite  literally.  In  this 
approach,  the  technical  scores,  corrected  for  evaluated  risk,  are  converted  to  a  standard  quantitative 
score  and  divided  by  a  differential  life-cycle  cost.  The  best  features  of  best  value  and  least  overall 
cost  criteria  are  both  used  in  making  the  source  selection.  The  highest  resulting  score  is  selected.  It 
is  unnecessary  to  disqualify  bidders  as  the  technical  scores  for  unqualified  bidders  will  be  too  low  to 
be  competitive.  This  method  is  relatively  easy  to  use  and  always  applicable  to  NDI  acquisitions  of 
all  types.  Although  somewhat  more  labor  intensive  to  establish,  this  approach  works  very  well  for 
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all  forms  of  NDI.  It  is  the  preferred  method  for  large,  complex  systems,  for  major  modifications,  and 
for  Integrated  NDI. 

The  solicitation  is  an  ideal  opportunity  to  obtain  information.  Some  of  the  information  that  should 
be  considered  include  the  following  elements; 

•  Performance  characteristics,  including  specifications  and  standards,  ranges  of  operation,  physi¬ 
cal  properties,  environmental  performance,  and  installation  characteristics. 

•  Supportability  characteristics,  including  supplier  support,  repair  costs,  and  upgrade  characteris¬ 
tics. 

•  Test  data,  including  results,  data,  methodology/procedures,  standards,  and  setups. 

•  Quality  data,  including  defect  metrics,  warranty  data,  scrap  rates,  rework  rates,  repair  rates, 
repair  turnaround  rates,  quality  implementation  plans,  and  customer  compliments/complaints. 

•  Recommended  lists  of  repair  items  and  consumables,  including  range  and  depth  of  support. 

•  Recommended  support  and  test  equipment. 

•  Recommended  plans  to  ensure  availability  of  spares  and  surge  capabilities  in  case  of  mobiliza¬ 
tion. 

•  Identification  of  proprietary  rights,  limited  rights  in  data,  and  patent  rights. 

•  Recommended  documentation,  manuals,  and  training  materials. 

•  Proposed  training  support. 

•  Proposed  warranty  provisions,  restrictions,  costs,  and  procedures. 

•  Health  and  safety  certifications. 

•  Availability  of  contractor  engineering  technical  services. 

•  Information  required  for  the  evaluations  (Section  L). 

•  Alternative  proposals. 

Some  of  this  information  can  be  obtained  during  market  surveys,  through  “sources  sought”  solicita¬ 
tions,  and  the  process  of  gathering  industry  comments  prior  to  a  formal  procurement  solicitation. 
Other  information  may  not  be  needed.  Note  that  most  of  this  information  is  related  to  the  sustain¬ 
ability  of  the  resulting  system.  The  information  may  be  required  if  the  sustainability  plans  have 
options  that  can  be  negotiated  during  the  procurement.  If  the  decisions  have  been  made  so  that  no 
options  exist,  the  data  should  be  supplied  in  accordance  with  the  CDRL.  All  of  the  information 
obtained  needs  to  be  assessed  against  the  needs  of  the  ultimate  users  and  maintainers,  especially  the 
potential  impact  on  availability  of  the  system  in  a  deployed  status  (see  SUPPORT  PLANNING). 

TECHNICAL  EVALUATION  CRITERIA  TIPS 

A  good  technical  evaluation  is  critical  to  any  source  selection  for  the  procurement  of  complex 
items.  Good  evaluations  are  structured  to  maintain  strict  fairness  to  all,  to  thoroughly  demonstrate 
the  proposer’s  understanding  of  the  requirements,  to  assess  the  capabilities  to  perform,  and  to  docu¬ 
ment  the  information  used  to  make  the  source  selection.  To  support  these  goals,  a  number  of  best 
practices  can  be  applied; 
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•  Prioritize  and  segment  the  requirements  into  related  groups. 

•  Form  evaluation  teams  of  technical  experts  for  each  requirements  group. 

•  Use  a  technical  evaluation  form  tailored  to  each  criterion  and  set  up  to  capture  narrative  evalu¬ 
ations  of  the  strengths  and  weaknesses  of  each  proposal.  Evaluators  should  have  to  explain 
why  the  criterion  is  being  graded  a  particular  way. 

•  Instruct  the  proposers  (Section  L)  to  format  the  proposals  consistently  with  the  grouping  of  the 
requirements.  (A  separate  proposal  volume  for  each  major  requirements  group.)  Include  a 
page  count  limitation,  either  by  requirements  group  or  overall. 

•  Include  an  assessment  of  the  risk  with  each  evaluation  element. 

•  Provide  descriptors  of  the  evaluation  grade  levels  that  include  quality  of  performance  offered 
and  risks  that  are  non-ambiguous.  Do  not  allow  too  many  gradations  of  evaluation,  but  make 
clear  distinctions  between  evaluation  levels. 

Evaluations  and  grading  systems  can  be  structured  in  many  ways  to  accomplish  the  goals  of  the  tech¬ 
nical  evaluation.  A  major  key  is  to  set  up  this  structure  so  that  it  is  easy  for  evaluators  to  be  consis¬ 
tent  in  their  assessments.  Also,  it  is  important  to  set  up  the  requirements  groupings  so  that  they  are 
comprehensive  yet  without  too  much  detail  that  dilutes  the  efforts  of  either  the  proposers  or  the  eval¬ 
uators.  A  good  approach  here  is  to  request  the  proposer  to  show  how  the  proposed  approach  will 
meet  a  required  figure  of  merit  that  describes  some  block  of  requirement  capabilities.  These  tips 
apply  whether  the  acquisition  is  NDI  or  not. 

Because  NDI  is  pre-existent,  a  technical  evaluation  may  include  several  things  that  would  not  be 
possible  for  developmental  acquisitions.  These  include  the  following: 

•  A  live  demonstration  of  the  system  capability  (or  some  significant  portion  for  integration 
NDI).  This  can  include  software  running  on  hardware,  live  communications,  set-up/tear-down 
demonstrations,  and  anything  else  identified  as  a  key  system  capability.  Real  operational  capa¬ 
bilities  in  real  environments  are  desired  in  demonstrations,  but  the  proposer  must  not  be 
required  to  make  unreasonable  investments  that  cannot  be  compensated  unless  the  contract  is 
won.  One  approach  is  to  conduct  an  initial  evaluation  to  prescreen  proposals,  and  then  to  pay 

a  fixed  fee  to  all  qualified  proposers  to  conduct  a  demonstration. 

•  Historical  information  that  demonstrates  the  capability  to  perform  at  low  risk.  This  can 
include  quality  and  reliability  information. 

•  Supplier  process  and  procedure  information  that  demonstrates  both  capability  and  maturity  in 
performing  the  required  tasks.  This  element  is  particularly  critical  for  software  requirements. 

•  Examination  of  actual  technical  data  to  be  delivered,  such  as  technical  manuals,  drawings. 
Interface  Description  Documents,  etc.  These  documents  are  loaned  for  purposes  of  the  evalua¬ 
tion  and  only  delivered  by  the  selectee. 

Proposers  of  NDI  should  be  encouraged  to  provide  alternate  proposals  of  new  or  upgraded  products 
that  may  have  been  undisclosed  to  the  market  survey.  The  technical  evaluation  should  evaluate  these 
alternatives  as  if  they  were  separate  proposals,  neither  adding  nor  detracting  from  the  proposers  main 
proposal. 
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TESTING  NDI 


All  acquisition  programs  must  perform  testing.  Standard  tests  include  operational  evaluations, 
technical  evaluations,  production/factory  acceptance  tests,  first  article  tests,  quality  conformance 
tests,  workmanship  screens,  safety  certification  tests,  and  reliability/maintainability  tests.  There  may 
also  be  human  factors  evaluations,  and  specialized  tests  for  software,  materials,  processes,  and  parts. 
These  tests  evaluate  the  system  and  its  components’  ability  to  meet  all  of  the  system  requirements. 

Of  course,  there  are  a  variety  of  developmental  tests  within  development  acquisitions  that  do  not 
apply  to  NDI  acquisitions.  All  of  the  major  forms  of  testing  apply  to  NDI  acquisitions,  but  the  char¬ 
acter  of  the  tests  may  change.  For  instance,  most  production  acceptance  testing  will  conform  to  the 
accepted  supplier  practices  for  COTS/ROTS  rather  than  being  specified  in  procurement  specifica¬ 
tions.  On  the  other  hand,  special  screens  and  documentary  tests  may  be  required  in  an  NDI  acquisi¬ 
tion  that  would  not  be  encountered  in  developmental  acquisitions.  Also,  special  tests  may  be  required 
for  certain  types  of  NDI  and  not  for  others.  Most  of  the  test  issues  associated  with  NDI  are  summa¬ 
rized  in  table  3. 


Table  3.  Test  and  evaluation  summary. 


Test  Requirement 

NDI  Advice  or  Requirement 

Operational  Evaluation 
(OPEVAL) 

Required  prior  to  Milestone  3.  May  be  combined  with  source  selection 
process  with  appropriate  planning  for  true  _OTS  systems. 

Added  time  required  for  non_OTS  systems. 

Technical  Evaluation 
(TECHEVAL) 

Required  prior  to  OPEVAL.  Should  be  combined  with  source 
selection  process  where  possible.  (Do  not  confuse  TECHEVAL  with 
the  technical  evaluation  conducted  as  part  of  the  source  selection 
process.) 

First  Article  Tests 

Only  required  when  there  is  a  true  first  article — especially  for  MILOTS, 
NewCOTS,  major  modifications,  and  integrated  NDI.  Should  also  be 
applied  for  NDS  and  most  modifications. 

Production  Acceptance  Tests 

Normally  not  required  beyond  established  and  accepted  supplier 
production  tests,  except  for  MILOTS.  (Test  protocols  can  be  reviewed 
and  accepted  in  accordance  with  IS09000  series  practices.)  May  be 
required  for  some  integrated  NDI  and  some  major  modifications.  May 
be  required  for  limited  duration  on  productions  of  NewCOTS. 

Quality  Conformance  Tests 

Normally  not  required  except  for  some  MILOTS,  Integrated  NDI,  and 
some  major  modifications. 

Workmanship  Screens 

Normally  Environmental  Stress  Screening  required  for  hardware  items 
being  subjected  to  a  more  severe  environment  than  that  for  which  the 
equipment  was  originally  designed  and  produced.  This  includes  most 
Navy  shipboard  environments.  May  be  applied  for  applications 
requiring  especially  high  reliability.  May  be  required  for  NewCOTS. 

Foreign  Capability  Evaluation 

Applied  to  FME  and  FOOTS  items  as  a  form  of  market  survey  screen. 
May  be  used  to  equalize  foreign  candidates  with  domestic  candidates 
for  source  selection  purposes. 

Supplier  Demonstration  Tests 

Used  as  part  of  the  technical  evaluation  portion  of  the  source  selection 
process,  where  applicable. 

Interoperability  Tests 

Required  at  each  system  level  of  complexity.  Integrated  at  the  top 
levels  with  TECHEVAL  and  other  system  tests.  Needed  at  lower 
levels  to  maintain  surveillance  of  interface  performance  throughout  the 
system  life.  Conducted  to  the  configuration  item  level. 
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Table  3.  Test  and  evaluation  summary.  (Continued) 


Test  Requirement 

NDI  Advice  or  Requirement 

Human  Factors  Demonstrations 

May  be  performed  during  the  market  survey  or  during  the  source 
selection  process.  Should  be  required  for  all  systems  having  a 
significant  or  new  operator  interface  requirement  and  all  systems 
having  complex  maintenance  tasks. 

Safety  Certification  Tests 

Normally  conducted  by  the  supplier  for  NDI.  May  be  required  for 
integrated  NDI. 

Reliability  Tests 

Heliability  Screen  may  be  required  to  supplement  other  available 
information  and  to  increase  confidence  in  reliability  data  for  making 
support  decisions. 

Maintainability  Demonstrations 

Not  normally  required  for  NDI,  but  may  be  required  in  preparation  for 
OPEVAL  for  maintenance-intensive  systems  at  the  organizational 
level  and  for  vital  system  verification  of  operational  availability 
requirements. 

Materials  Tests 

Not  required  unless  new  materials  are  being  introduced  or  material 
substitutions  have  been  made  to  modify  NDI  to  meet  application 
requirements  and  the  substituted  materials  have  not  been  previously 
qualified. 

Process  Tests 

Suggested  when  quality  and  reliability  requirements  are  high  and  new 
processes  are  introduced.  Normally  conducted  by  supplier  with 
notification  to  acquirer  under  IS09000  series  procedures.  May  be 
required/negotiated  for  some  NDS  acquisitions. 

Parts  Tests 

Conducted  by  supplier  as  required.  No  notification  required  to 
acquirer. 

NDI  Screening  Tests 

Should  be  conducted  as  part  of  market  survey  to  close  gaps  between 
market  requirements  and  standards  and  military  application 
requirements.  Usually  involve  combined  environment  testing 
(especially  shock,  vibration,  temperature,  and  humidity), 
electromagnetic  compatibility  testing  (including  power  system 
interfaces),  and  special  performance  requirements.  Tests  may  also  be 
conducted  to  develop  interface  control  information  outside  of  existing 
interface  standards. 

Capability  Demonstration 

May  be  required  as  a  part  of  the  source  selection  process  for  true  NDI 
elements.  May  vary  from  a  full  system  demonstration  in  operational 
environments  to  demonstrations  of  key  subsystem  items  at  supplier’s 
facilities.  Consider  feasibility  of  combining  other  program  tests  (such 
as  OPEVAL  and  TECHEVAL)  into  the  demonstration. 

Regression  Tests 

Normally  required  tor  revised  NDS,  conducted  by  acquirer’s  Software 
Support  Activity. 

Please  note  that  table  3  is  not  intended  to  be  exhaustive;  however,  it  covers  the  most  commonly 
encountered  tests.  A  competent  system  test  director  should  be  employed  on  the  acquisition  team  to 
determine  any  special  test  requirements  as  well  as  to  tailor  individual  test  requirements,  review  sup¬ 
plier  test  documentation,  witness  supplier  tests,  and  supply  advice  to  the  project  manager  on  test 
issues. 
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RELIABILITY  ISSUES 


Three  levels  of  reliability  must  be  considered  and  reconciled  for  any  acquisition:  system,  product, 
and  component.  Because  of  the  character  of  NDI,  the  emphasis  tends  to  shift  from  component  reli¬ 
ability  (high  emphasis  in  developments)  to  system  reliability.  Good  system  design  practices  will 
incorporate  reliability  issues  in  meeting  the  operational  requirements,  but  NDI  acquisitions  also 
require  many  of  the  solutions  to  be  accomplished  at  the  system  level  as  well.  This  is  because  the  sys¬ 
tem  designer  is  “stuck”  with  the  product  and  component  reliability  of  the  NDI  once  a  source  selec¬ 
tion  is  made.  The  goal  of  the  reliability  program  remains  to  maximize  both  operational  availability 
and  dependability  for  the  system  users. 

Component  reliability  emphasizes  the  selection  of  high-quality  components,  good  thermal  design 
practices,  good  derating  practices,  and  avoiding/mitigating  known  component  failure  modes  through¬ 
out  the  product  design.  NDI  projects  do  not  have  control  over  these  elements;  however,  that  does  not 
prevent  the  system  reliability  experts  from  evaluating  the  design  of  the  products.  A  quick  scan  of  the 
product  design  documentation  will  often  reveal  the  integrity  of  the  product  design  in  its  application 
of  the  components  While  this  information  may  not  be  sufficient  to  establish  support  plans,  it  can  be 
crucial  to  resolving  the  confidence  levels  of  information  from  other  sources  and  in  identifying  poten¬ 
tial  problems. 

Many  MIL  development  programs  result  in  low-volume  productions.  This  means  that  they  depend 
on  increasing  the  component  application  reliability  levels  through  parts  control  programs  and  other 
means.  MILOTS  and  FME  items  should  have  extensive  component  application  information  avail¬ 
able  for  review.  Most  COTS,  ROTS,  and  FCOTS  use  industrial  best  practices  and  do  not  emphasize 
component  reliability.  Reliability  issues  are  resolved  at  the  product  level  through  continuous  product 
improvements.  For  instance,  the  higher  production  rates  of  COTS  items  usually  justify  the  use  of 
application  specific  integrated  circuit  (ASIC)  technology,  and  other  inherently  reliable  technologies, 
that  may  not  be  affordable  to  many  MIL  applications.  COTS  suppliers  have  an  economic  incentive 
to  make  their  products  reliable.  Low  product  reliability  will  cut  profits  and  market  share.  Mature 
COTS  products  will  have  good  field  data  to  support  the  reliability  assessments;  however,  NewCOTS 
will  lack  this  information.  A  more  detailed  component-level  reliability  assessment  is  warranted  for 
NewCOTS. 

Integrated  NDI  and  modifications  should  normally  have  component-level  application  guidelines 
(and  software  development  plans  [SDP],  as  appropriate)  to  promote  good  system  reliability  when  all 
of  the  system  elements  are  integrated  together.  Unlike  development  programs,  integrated  NDI  need 
not  have  extensive  reliability  analyses  and  reviews  as  long  as  the  approved  guidelines  are  mutually 
accepted  and  implemented  by  the  supplier  and  acquirer. 

Product  reliability  is  driven  by  the  component  application  reliability,  by  production  quality,  and  by 
economic  tradeoffs  between  initial  costs  and  life-cycle  support  costs.  Since  most  commercial  items 
have  a  very  short  product  life  (5  to  6  years  is  typical)  compared  to  military  system  lives  (typically  20 
to  30  years),  the  reliability  factors  driving  life-cycle  support  costs  are  not  as  important  in  this  trade¬ 
off.  This  is  not  to  say  that  COTS  items  are  inherently  less  reliable  than  “equivalent”  military  items. 

In  fact,  the  opposite  has  been  found  to  be  true  in  the  majority  of  studies  comparing  like  products 
from  these  diverse  cultures.  Most  “militarized”  products  lack  sufficient  production  requirements  to 
even  begin  to  resolve  all  of  the  production  process  elements  that  improve  product  quality  and  reli¬ 
ability,  and  component  reliability  can  seldom  be  maximized.  Commercial  items  achieve  component 
reliabilities  an  order  of  magnitude  less  than  most  military  items,  yet  the  commercial  products  may 
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have  a  product  reliability  several  times  higher  than  the  military  product.  The  primary  difference  is 
the  contribution  of  high-volume  production  processes  that  compensate  for  the  lack  of  component 
reliability.  Some  commercial  markets  have  a  very  high  reliability  requirement  (medical  electronics, 
for  instance)  and  employ  high-reliability  components  in  combination  with  high-quality  processes  to 
produce  products  at  extremely  high  reliability  levels  (comparable  to  space  qualified  systems). 

System  reliability  rests  on  product  reliability  plus  the  system  architecture/system  partitioning  deci¬ 
sions.  Highly  reliable  systems  can  be  constructed  from  basically  unreliable  pieces,  at  a  price,  if  the 
underlying  reliabilities  can  be  adequately  characterized.  High  system  reliability  is  essential  to  highly 
effective  and  affordable  operational  availability  and  dependability.  On  the  other  hand,  poor  system 
designs  have  implemented  low  levels  of  reliability  even  though  component  reliability  was  maxi¬ 
mized.  In  NDI  systems,  system  reliability  design  is  crucial.  Since  the  product/component  reliabili¬ 
ties  are  fixed,  the  system  level  is  the  only  place  that  there  is  flexibility  to  meet  the  operational  reli¬ 
ability  objectives. 

A  major  problem  for  the  system  reliability  designer  is  the  lack  of  high-confidence  information  on 
the  NDI  components.  The  reliability  must  be  allocated  from  the  system  level  to  the  subsidiary  levels 
in  a  way  that  the  allocated  reliability  is  compatible  with  the  realized  product  reliability  at  that  level. 
For  a  given  NDI,  there  may  be  an  order  of  magnitude  difference  in  failure  rates  between  the  low-end 
and  top-end  suppliers.  Furthermore,  field  failure  rate  data,  test  data,  warranty  data,  and  analysis  pre¬ 
dictions  may  disagree  by  a  factor  of  10.  Such  large  variations  are  intolerable  for  support  planning 
purposes  unless  all  of  the  data  indicates  a  reliability  substantially  higher  than  the  expected  service 
life  of  the  product,  and  the  product  population  is  relatively  small.  The  system  designer  must  realize 
that  none  of  these  sources  of  information  is  “wrong”  but  that  the  variations  are  artifacts  of  the  low 
confidence  levels  that  are  sometimes  associated  with  the  various  sources.  Some  effort  is  required  to 
resolve  the  problems  created  by  the  low  confidence  levels  that  may  involve  added  market  investiga¬ 
tions  and  testing. 

How  does  the  system  designer  deal  with  this  problem?  Field  data  (including  warranty  data)  is  gen¬ 
erally  the  highest  confidence  level  of  all  sources  of  data;  however,  this  is  not  true  for  immature  or 
low-population  systems,  such  as  NewCOTS  and  many  MILOTS  and  FME.  Field  data  may  also  be 
for  application  environments  that  do  not  accurately  reflect  the  stresses  of  the  new  system  application, 
and  adjustments  between  the  environments  may  have  a  low  confidence  in  themselves.  Nevertheless, 
field  data  is  used  as  a  source  of  preference  when  the  product  being  considered  is  mature.  Testing  is 
the  next  most  reliable  source;  however,  high-reliability  systems  virtually  preclude  sufficient  testing 
programs  to  generate  reliable,  high-confidence  data.  Fortunately,  field  data  and  test  data  can  often  be 
combined  for  analysis  purposes  under  the  right  conditions.  If  the  testing  is  conducted  to  stress  the 
product  with  a  combined  environment  meeting  or  exceeding  the  intended  application,  the  resulting 
failures  can  be  analyzed  to  deterrmne  if  any  new  failure  modes  have  been  exposed  due  to  the  envi¬ 
ronmental  stress,  and  the  added  test  time  can  be  combined  with  known  field  use  time.  In  addition,  a 
thermal  survey  can  be  added  to  the  testing;  the  thermal  survey  is  a  quick  and  inexpensive  test  that 
provides  a  high  degree  of  information  about  the  reliability  design  and  field  performance  of  the  equip¬ 
ment.  Similarly,  stress  testing  software  can  yield  valuable  information  that  can  verify  other  sources 
of  information  and  help  to  raise  the  confidence  to  acceptable  levels.  Reliability  predictions  are  the 
least  reliable  source  of  information  by  themselves;  however,  they  can  contribute  significantly  to 
resolving  data  differences  between  field  data  extracted  from  different  sources  or  from  different 
application  environments.  Also,  predictions  can  give  valuable  insights  into  potential  disagreements 
betwepn  field  data  and  test  data.  Predictions  are  quite  useful  in  determining  how  much  test  time  may 
be  required  to  resolve  confidence  levels.  Since  test  time  is  costly,  the  predictions  may  be  essential 
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for  project  planning  with  sufficient  lead  time  to  obtain  the  needed  resources.  Finally,  predictions  are 
often  helpful  in  giving  indicators  about  the  effects  of  production  quality.  If  there  are  substantial  dif¬ 
ferences  between  two  sources  of  supply  in  quality,  there  should  be  an  observable  difference  in  field 
reliability  that  should  agree  with  predictions  within  a  factor  of  two.  It  is  useful  for  the  system 
designer  to  obtain  as  much  data  from  potential  suppliers  as  possible,  including  the  supplier’s  predic¬ 
tions.  Each  source  of  information  adds  a  perspective  and  fills  in  a  piece  of  the  overall  puzzle.  A  bit 
of  information  may  not  be  of  high  confidence  in  itself  nor  answer  the  ultimate  system  reliability 
questions,  but  the  accumulated  bits  allow  the  system  designer  to  get  good  enough  information  in  total 
to  continue  the  system  planning  with  confidence. 

One  of  the  critical  issues  of  system  reliability  design  is  the  single  point  failure  (SPF).  In  general, 
system  designers  try  to  avoid  SPFs.  Even  if  one  is  successful  in  constructing  a  top-level  architecmre 
avoiding  single  points  of  failure,  lower  level  SPFs  can  be  introduced  by  NDI  components.  This  is 
especially  true  when  the  item  was  designed  for  a  benign  commercial  application  but  is  being  applied 
to  a  mission-critical  application.  Awareness  of  the  SPF  can  allow  the  system  designer  to  employ 
high-level  redundancy,  automatic  backup,  or  operational  sparing  to  achieve  the  requisite  levels  of 
system  reliabilitv  and  availability.  Among  the  crucial  elements  to  be  considered  are  power  supplies. 
Power  supplies  are  often  SPFs  for  developed  systems,  but  they  are  almost  always  SPFs  in  COTS  sys¬ 
tems.  System  designer,  beware! 

During  the  system  partitioning/market  survey,  the  system  designer  should  obtain  the  following 
information  elements,  as  a  minimum,  if  possible; 

•  The  supplier  reliability  design  program  (not  merely  marketing  brochure  claims,  but  docu¬ 
mented  es  idenee).  including  derating  criteria,  thermal  analyses,  predictions,  and  shock/vibra¬ 
tion  analysis 

•  Product  MTBF-  based  on  field/warranty  data. 

•  Target  design  en\  ironment  for  the  product. 

•  Failure  Modes.  Ftfeeis.  and  Criticality  Analysis  results. 

•  Employment  ot  Lnvironmental  Stress  Screening,  part  selection/qualification  programs,  critical 
part  inspections,  and  other  workmanship/quality  provisions. 

•  System  and  subs\  stem  reliability  testing  (procedures  and  results). 

The  reliability  design  is  heavily  interdependent  with  the  supplier  quality  practices.  An  excellent  reli¬ 
ability  design  can  be  compromised  by  poor  quality  practices.  Excellent  quality  practices  can  par¬ 
tially  compensate  for  a  poor  reliability  design.  Ideally,  an  excellent  reliability  design  will  be  imple¬ 
mented  with  an  excellent  quality  program  as  well. 

OVERALL  SYSTEM  RELIABILITY  ISSUES 

While  most  reliabilitc  issues  have  traditionally  rested  on  hardware  failure  modes  and  effects,  sys¬ 
tems  consist  of  hardware,  software,  and  operators/maintainers  in  interaction  with  each  other.  There¬ 
fore,  system  reliability  must  consider  all  of  these  factors  both  independently  and  in  combination. 

Hardware  reliability  issues  are  primarily  driven  by  physical  factors.  Understanding  the  physics  of 
failure  allows  suitable  steps  to  be  taken  in  system  design,  product  design,  production,  quality  control, 
process  selection,  material  selection,  and  so  forth,  to  achieve  very  high  levels  of  reliability  in  the 
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most  rigorous  levels  of  application  stress.  Traditional  reliability  disciplines  and  tools  aid  in  accom¬ 
plishing  high  reliability  in  an  affordable  way. 

Software  has  increasingly  assumed  greater  burdens  of  system  functionality.  Since  software  does 
not  have  a  true  physical  nature,  it  is  not  subject  to  failure  in  the  same  sense  as  hardware.  Neverthe¬ 
less,  software  may  contain  latent  defects,  may  be  executed  in  unintended  ways  when  interacting  with 
other  software  or  hardware  elements,  or  may  not  be  designed  to  deal  with  hardware  faults  and  human 
errors.  Ideally,  software  should  be  designed  for  all  potential  interface  and  error  conditions  as  well  as 
performing  all  intended  functions  perfectly.  In  practice,  this  is  not  usually  possible  even  in  the 
abstract  because  software,  especially  NDS  and  reuse  software,  will  be  used  in  applications  and  envi¬ 
ronments  that  could  not  have  been  anticipated  by  any  designer.  As  it  turns  out,  the  conditions  that 
lead  to  a  software-driven  system  failure  have  a  statistical  behavior  that  is  not  too  different  from  hard¬ 
ware  failures,  so  the  mathematics  used  at  the  system  level  for  analyzing  reliability  is  well  understood. 
Considerable  research  has  been  conducted  since  the  early  1980s  (and  continues)  on  software  reliabil¬ 
ity  issues.  The  key  for  the  system  designer  is  to  obtain  software  development  quality  metrics  and 
field  quality  metrics,  to  conduct  rigorous  software  capability  evaluations  tailored  to  the  intended 
application  and  computing  environment  with  appropriate  system  stresses  included,  to  construct 
appropriate  application  process  unit  metrics,  and  to  conduct  well-instrumented  system  integration/ 
system  interoperability  tests.  While  these  elements  should  be  found  in  any  well-designed  system 
acquisition,  they  are  even  more  important  in  NDI  acquisitions  because  the  software  products  are  pre¬ 
existent  and  must  be  characterized  for  the  new  system  application.  These  steps  in  combination  allow 
the  development  of  an  accurate  picture  of  system  behavior  in  the  field,  even  when  the  fielded  config¬ 
urations  are  changing  rapidly  (a  characteristic  of  NDI-based  systems).  A  wide  variety  of  models 
exist  to  assess  software  reliability  and  to  combine  the  knowable  information  about  the  software  being 
considered  for  inclusion  in  a  system.  These  models  allow  the  development  of  information  critical  to 
making  support  decisions,  planning  SSA  funding  levels,  and  performing  the  needed  tests  cost  effec¬ 
tively. 

The  third  component  of  system  reliability  is  the  human  operator  or  maintainer.  Humans  are  sub¬ 
ject  to  making  errors,  especially  under  the  stress  of  combat.  Human  error  also  has  a  random 
character  that  is  similar  in  its  statistics  to  hardware  failures;  therefore,  system-level  analysis  tools 
exist  to  include  human  performance  in  overall  system  reliability.  Not  surprisingly,  human  operators 
are  often  found  to  be  single  point  failures  in  system  reliability  analyses.  The  system  designer  must 
balance  human  factors  design  (see  HUMAN  FACTORS)  and  training  requirements  (see  TRAIN¬ 
ING)  in  order  to  achieve  high  levels  of  system  performance  in  the  field  at  an  affordable  price.  A  key 
element  of  performing  this  task  is  the  definition  of  meaningful  metrics  of  human  performance  as  a 
part  of  the  system  that  can  sense  human  stress  conditions  (such  as  combat  performance  or  the  frustra¬ 
tion  level  after  making  an  error  or  the  confusion  when  presented  with  unexpected  conditions).  In 
NDI  systems,  the  human  factors  and  training  elements  are  severely  constrained,  so  the  allocation  of 
system  failures  must  first  consider  these  constraints,  resulting  in  higher  requirements  for  the  software 
and  hardware  products. 


The  key  to  understanding  all  system  reliability  issues  is  to  recognize  that  stresses  promote  failures. 
The  stresses  promoting  failures  are  dilferent  for  hardware,  software,  and  humans,  but  the  system 
designer  can  identify,  analyze,  and  quantify  these  stresses.  If  the  stresses  are  known,  it  is  much  eas¬ 
ier  to  determine  appropriate  actions,  such  as  redesign,  reallocation  of  requirements,  or  encapsulation 
(isolating  a  system  element  from  the  stress),  that  result  in  affordable  systems  that  perform  adequately 
with  acceptable  operational  risk.  It  is  essential  to  obtain  appropriate  expertise  to  participate  in  these 
system  reliability  issues,  especially  as  they  bear  on  the  market  survey  activities  and  screening 
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activities  supporting  the  overall  acquisition.  This  level  of  system  expertise  should  be  retained  in  the 
life-cycle  management  activities  after  the  system  is  fielded  in  order  to  promote  effective  decisions  in 
the  replacement  decisions  made  during  the  support  of  the  system. 

Usually,  all  of  the  factors  needed  to  evaluate  system  performance  do  not  occur  until  system 
integration  tests  or  TECHEVAL.  NDI  acquisitions  are  often  so  short  that  no  prior  testing  will  be  fea¬ 
sible  prior  to  these  high-level  system  tests;  therefore,  the  acquiring  agent  should  expect  that  some 
surprises  will  occur  because  the  accumulated  risk  can  be  quite  high  at  the  system  level.  As  long  as 
these  surprises  are  anticipated  and  the  tests  are  instrumented  with  appropriate  measures  of  perfor¬ 
mance  (for  hardware,  software,  humans,  and  the  total  system)  and  some  level  of  effort  is  planned 
(and  funded)  to  handle  minor  contingencies,  the  project  can  proceed  and  attain  success.  All  of  these 
elements  will  be  required  in  order  to  obtain  high  user  satisfaction  with  the  fielded  system. 


QUALITY  PROVISIONS 

Quality  is  the  ability  to  meet  requirements  generated  by  implemented  processes  and  procedures 
throughout  the  product  life  cycle  (concept,  design,  production,  and  support).  At  least  three  different 
perspectives  on  quality  can  be  combined  to  create  an  overall  quality  profile  for  NDI  products;  these 
are; 

•  Process  assurance. 

•  Product  assurance. 

•  Market  assurance. 

Process  assurance  focuses  on  maintaining  excellent  processes  in  the  creation  and  management  of  a 
product  with  the  underlying  assumption  that  good  processes  create  good  products.  This  assumption 
is  borne  out  by  field  experience.  Product  assurance  imposes  procedures  to  contribute  to  overall  qual¬ 
ity,  such  as  inspections,  tests,  and  audits.  Many  of  the  good  processes  chosen  in  a  process  assurance 
perspective  will  contain  procedures  from  the  product  assurance  perspective  tailored  to  the  process. 
Market  assurance  combines  company  practices  with  market  dynamics  to  maintain  a  sufficient  quality 
perspective  to  remain  cost  competitive  and  reputation  competitive  in  a  particular  product  market. 
Market  assurance  may  adopt  process  assurance  and  product  assurance  practices,  but  it  is  also  heavily 
influenced  by  competition  factors  in  the  marketplace.  Each  supplier  maintains  a  different  market 
assurance  quality  perspective,  and  this  perspective  may  vary  from  product  to  product  even  by  the 
same  supplier.  The  various  quality  standards  are  an  attempt  to  overcome  the  uncertainties  associated 
with  market  assurance. 

The  IS09000  series  of  quality  standards  represent  a  worldwide  attempt  to  provide  stability  in 
product  quality.  This  series  emphasizes  process  assurance  activities.  There  are  procedures  for 
obtaining  an  IS09000  certification  that  testify  that  a  supplier  has  formal  quality  processes  in  place 
that  meet  at  least  the  minimum  standards.  In  addition  to  the  minimum  standard  certification,  the 
IS09000  series  provides  for  a  customer-supplier  dialog  to  allow  the  customer  to  obtain  customized 
quality  activities  for  the  products  or  services  delivered.  Among  companies  maintaining  IS09000 
certifieation,  there  is  still  some  variability  in  quality,  but  the  quality  levels  of  individual  products  are 
more  predictable  and  uniform  than  for  companies  driven  by  market  assurance  alone.  Part  of  the  cer¬ 
tification  requirements  include  maintenance  of  basic  quality  records  that  can  be  made  available  to  the 
potential  customer  base  so  that  customers  can  accomplish  an  independent  quality  assessment  prior  to 
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buying  a  product  or  service.  These  records  should  be  accessed  by  the  system  designer  anticipating 
use  of  NDI  as  a  fundamental  part  of  the  market  survey. 

The  system  designer  must  also  analyze  the  system  partitions  to  determine  where  exceptional  levels 
of  quality  may  be  required  and  justified.  At  least  the  following  system  elements  must  be  considered 
for  special  attention: 

•  Mission-critical  elements  (even  if  they  are  not  designated  as  mission  critical). 

•  Single  point  failure  elements. 

•  Any  element  that  has  a  high  consequence  of  failure  (safety  critical,  mission  failure  critical). 

•  High-value  support  items. 

System  elements  that  justify  special  quality  attention  will  also  require  special  attention  for  support, 
testing,  and  reliability  assessment. 

Good  quality  practices  by  potential  suppliers  should  by  complemented  by  good  quality  practices 
by  the  acquiring  activity.  Acquiring  activity  quality  practices  should  include  the  following  measures: 

•  Good  system  design  processes  and  practices. 

•  Reliability  program  support. 

•  Test  program  support. 

•  Internal  qualii\  expertise  to  work  with  potential  suppliers. 

•  Good  market  analyses  and  surveys. 

•  Cost-benefit  anal\  sis  support. 

System  expertise  must  he  assembled  into  an  acquisition  team  and  effectively  employed  throughout 
the  development  of  s\  stem  requirements,  procurement,  testing,  fielding,  and  support. 

Since  many  sources  ot  NDI  are  driven  by  market  quality  assurance,  it  is  often  practical  to  add  a 
post-production  qualii>  screen  to  provide  some  stability  to  the  product  being  acquired.  The  practice 
of  lot  acceptance  and  periodic  acceptance  tests  is  normally  redundant  to  supplier  production  quality 
practices  and  should  he  heavily  tailored  for  the  various  COTS,  ROTS,  FCOTS,  and  NDS.  Neverthe¬ 
less,  a  limited  periodic  acceptance  test  can  be  useful  to  document  the  quality  characteristics  of 
dynamic  products  being  used  in  a  critical  system  application.  A  workmanship  screen  is  often  helpful 
for  hardware  items.  For  years,  the  Navy  has  successfully  and  cost  effectively  applied  Environmental 
Stress  Screening  (ESS  i  (MlL-STD-2164)  to  COTS  procurements  as  a  matter  of  policy.  Items  fail¬ 
ing  the  ESS  can  be  reu  orked  and  retested.  ESS  can  be  applied  by  the  supplier  prior  to  final  system 
assembly  for  large  systems.  For  small  systems  (black  box  systems),  ESS  can  be  applied  after  deliv¬ 
ery  and  failed  items  returned  under  warranty.  In  all  cases,  the  system  acquirer  must  judiciously  tailor 
quality  practices  in  order  to  achieve  the  system  goals  cost  effectively.  The  most  cost-effective  means 
of  achieving  quality  is  through  the  process  assurance  obtained  on  high-rate  production  lines,  but  this 
is  sensitive  to  market  forces.  The  steps  needed  to  tailor  quality  practices  may  differ  for  each  NDI, 
and  the  information  needed  for  effective  quality  management  must  often  be  obtained  through  the 
market  analysis  and  tailored  screen.  ° 

Quality  issues  are  critical  to  solid  system  planning.  Inappropriate  decisions  can  make  quality 
implementation  into  a  program  cost  driver,  especially  in  NDI  acquisitions.  In  general,  the  highest 
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quality  possible  should  be  obtained  from  the  suppliers  without  disturbing  the  supplier’s  best  prac¬ 
tices.  If  added  quality  is  needed  to  overcome  market  inadequacies,  it  is  usually  best  for  one  of  the 
acquisition  program’s  agents  to  take  the  necessary  steps  (tests,  inspections,  etc.)  and  to  return  defec¬ 
tive  items  under  warranty  provisions.  This  approach  obtains  high  quality  without  creating  cost  driv¬ 
ers  in  the  commercial  market.  On  the  other  hand,  some  markets  offer  high-quality  options  that 
greatly  exceed  DoD  requirements;  these  options  should  be  exercised  with  care  to  ensure  that  a  rea¬ 
sonable  price  is  being  paid  for  the  true  needs. 

CONFIGURATION  MANAGEMENT  (CM)  ISSUES 

Configuration  Management  (CM)  is  essential  to  any  project.  However,  several  CM  issues  are 
important  to  NDI  projects.  These  include  the  following: 

•  Interface  control. 

•  Level  of  control. 

•  Configuration  identification. 

•  Configuration  status  accounting. 

•  Change  control. 

All  of  these  issues  involve  modified  perspectives  and  procedures  when  dealing  with  NDI  versus 
development  items. 

Interface  control  is  essential  for  NDI  systems.  NDI  systems  are  usually  characterized  by  frequent 
systems  upgrades  performed  along  form,  fit,  function  system  partitions.  The  top-level  partitions 
must  be  thoroughly  documented  by  the  system  designers  so  that  the  system  function  is  not  dependent 
on  any  second-order  parameters  (unspecified  but  essential  for  operation  parameters).  This  may 
imply  specialized  characterization  testing  during  the  system  design  and  prior  to  the  writing  of 
specifications  for  procurement.  It  is  especially  important  to  thoroughly  characterize  and  document 
the  hardware-software  interfaces  that  may  exist.  Eventually,  one  of  the  major  tools  available  to  the 
life-cycle  support  agent  of  NDI  (and  especially  COTS)  is  the  interface  documentation  down  to  the 
level  of  support.  This  level  of  documentation  establishes  the  level  of  standardization  internal  to  the 
system  architecture.  Without  effective  interface  control,  modifications  cannot  be  managed  efficiently 
and  life-cycle  support  is  less  effective.  Interface  control  is  also  important  because  most  of  the  NDI 
items  are  outside  of  the  project’s  management  control,  so  control  of  the  interfaces  provide  the  main 
method  of  technical  control. 

It  is  also  important  to  establish  a  realistic  level  of  CM  control.  If  CM  is  imposed  at  a  too  detailed 
level  of  system  complexity,  the  project  will  pay  for  huge  amounts  of  unneeded  documentation,  and 
then  continue  to  pay  for  maintaining  that  documentation.  If  CM  is  imposed  at  a  too  high  level  of 
system  complexity,  insufficient  information  will  exist  to  maintain  interface  control,  resulting  in  an 
inability  to  cost  effectively  manage  changes,  modifications,  technical  upgrades,  and  life-cycle  sup¬ 
port.  System  support  decisions,  level-of-repair  decisions,  technology  infusion  plans,  and  the  exercise 
of  CM  control  by  the  project  office  (through  either  the  System  Design  Agent  or  the  life-cycle  support 
agent)  all  need  to  be  coordinated  and  maintained  consistently.  The  project  office  must  maintain 
some  level  of  CM  control,  but  also  realize  that  detailed  control  of  the  NDI  product  must  remain  with 
the  product  agent  (the  supplier  in  the  case  of  COTS/ROTS;  the  original  acquisition  office  in  the  case 
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of  MILOTS  and  FME).  A  firm  rule  of  thumb  cannot  be  dictated  for  modifications  and  integrated 
NDI,  but  it  is  likely  that  the  project  office  (through  its  agents)  will  desire  to  maintain  CM  control  of 
the  modifications.  Occasionally,  a  system  integration  agent  will  be  retained  to  maintain  CM  control 
of  integrated  NDI  throughout  its  life. 

Configuration  identification  is  a  fundamental  CM  function.  The  rule  of  thumb  for  NDI  is  to  iden¬ 
tify  any  NDI  product  as  a  configuration  item.  Since  the  configuration  item  level  is  generally  the 
level  at  which  most  documentation  requirements  are  specified,  this  rule  will  almost  always  result  in  a 
configuration  item  identification  that  is  consistent  with  the  interface  control  requirements  and  level 
of  CM  control  desired.  Whenever  possible,  only  unmodified  NDI  should  be  a  configuration  item 
and  any  modifications  should  be  separately  identified.  This  enables  a  stable  system  architecture  that 
isolates  the  rest  of  the  system  from  any  major  changes  that  may  occur  within  the  NDI  over  its  prod¬ 
uct  life.  This  makes  life-cycle  management  much  easier  and  more  effective  while  not  inhibiting  the 
independent  support  of  the  NDI  by  the  appropriate  agent. 

As  an  example  of  this  configuration  identification  scheme,  assume  the  desired  system  is  a  special¬ 
ized  application  running  on  a  Navy  TAC  workstation.  The  TAC  workstation,  POSIX  operating  sys¬ 
tem,  X- Windows  user  interface,  a  commercial  word  processor,  and  Oracle  database  would  all  be  NDI 
configuration  items.  Modifying  scripts  to  interface  through  X-Windows,  application  macros  for  the 
word  processor,  and  application  script  for  Oracle  would  each  constitute  a  configuration  item,  even 
though  some  of  these  items  might  be  very  small.  Custom  application  software  would  be  identified 
into  configuration  items  in  accordance  with  development  rules  of  thumb  (functional  items  not  to 
exceed  about  10,000  lines  of  source  code).  Such  a  system  is  very  typical  of  integration  NDI  systems. 

This  scheme  of  configuration  identification  has  been  found  to  be  extremely  effective  in  both  cost  and 
function. 

Configuration  status  accounting  is  probably  the  most  crucial  CM  function  in  most  systems  con¬ 
taining  significant  NDI  functionality.  It  is  particularly  important  to  maintain  good  status  accounting 
for  COTS  products.  NDI  product  baselines  may  change  frequently  and  irregularly.  This  can  result"" 
in  fielded  systems  built  to  the  same  product  architecture  still  being  different  from  each  other  in  both 
detailed  functionality  and  in  support  item  requirements.  Some  COTS-based  systems  have  hundreds 
of  fielded  configurations.  A  single  installation  site  may  have  several  different  configurations  repre¬ 
senting  the  evolution  of  the  system.  Although  the  differences  between  systems  may  not  be  very  sig¬ 
nificant,  it  is  absolutely  essential  to  maintain  good  status  accounting  down  to  the  level  at  which  the 
system  is  being  supported.  If  this  is  not  planned  and  executed,  the  fielded  systems  will  invariably 
“break”  because  of  some  revision  of  an  item,  and  the  problems  trying  to  troubleshoot  the  causes  will 
be  enormous.  Good  status  accounting  will  not  prevent  “breakage,”  but  it  will  allow  quickly  identify¬ 
ing  the  combination  of  items  that  do  work  versus  those  that  do  not  work,  resolving  the  causes  rather 
quickly.  Once  the  “A  doesn’t  interface  with  B  anymore”  type  problem  is  found,  the  solutions  are 
usually  well  defined  and  quickly  implemented.  Also,  NDI  may  be  available  from  multiple  sources 
with  the  same  form,  fit,  and  function”  but  different  internal  parts. 

An  infrastructure  for  configuration  status  accounting  can  be  established  almost  immediately  as  a 
system  is  partitioned  by  establishing  and  maintaining  a  work  breakdown  structure  (WBS)  down  to 
the  configuration  items  ultimately  identified  for  the  system.  In  a  good  system  design,  the  WBS  is 
stable  even  when  elements  internal  to  the  configuration  items  change.  A  framework  of  references 
can  be  designed  for  every  installation  and  planned  future  installation  using  this  scheme.  The  refer¬ 
ences  can  be  used  to  preload  databases  used  for  various  support  elements.  The  agent  with  confimra- 
tion  management  responsibilities  (the  Life-Cycle  Management  Agent  is  recommended),  can  then 
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matrix  all  installed  items  (hardware,  software,  documentation,  and  training  products)  against  these 
references.  Interface  data,  performance  metrics,  specifications,  and  other  information  can  also  be 
matrixed  to  this  reference  system  so  that  a  consistent  frame  of  reference  is  easily  maintained  for  each 
installation  that  spans  all  of  the  system  design  and  support  issues.  If  the  matrix  is  maintained  in  a 
suitable  technical  platform,  all  of  the  information  can  be  accessed  remotely  or  locally  using  web 
browser  technology. 

Change  control  is  quite  different  for  COTS  applications.  The  COTS  manufacturer  will  make 
changes  for  a  variety  of  reasons  without  notifying  customers.  Normally,  these  changes  will  result  in 
less  expensive  systems  with  better  quality  and  reliability  performance.  However,  occasionally  the 
changes  will  result  in  poor  performance  in  some  environments.  The  system  agent  must  then  decide 
what  to  do.  Ordering  a  product  design  change  is  usually  not  an  option,  and  almost  never  a  cost-ef¬ 
fective  option.  Enabled  by  good  status  accounting,  the  system  agent  can  determine  the  best  course  of 
action  including: 

•  Stay  with  an  older  version  of  the  NDI  product  (newer  is  not  always  better). 

•  Go  back  to  the  prior  version  of  the  product  that  worked,  if  available. 

•  Design  new  interfacing  components  to  readapt  the  new  NDI  version  into  the  system. 

•  Disqualify  the  NDI  source  and  switch  to  another  source  maintaining  the  same  standard  inter¬ 
face. 

•  Rearchitecture  the  system  to  another,  more  stable,  interface  standard. 

In  COTS  systems,  all  of  these  options  may  be  viable;  only  two  or  three  of  the  options  may  be  work¬ 
able  in  other  forms  of  NDI.  Acquisition  managers  and  systems  engineers  need  to  recognize  that  usu¬ 
ally  over  10  percent  of  a  developed  system  will  be  changed  annually  for  a  variety  of  reasons.  NDI 
systems  often  do  not  undergo  any  more  annual  change  than  this,  except  the  change  control  is  exer¬ 
cised  by  the  product  design  agent  rather  than  the  system  agent.  The  system  agent  is  still  maintaining 
change  control,  except  that  system  CM  change  control  is  being  exercised  at  a  level  above  the  NDI 
product  level.  Trying  to  impose  lower  levels  of  change  control  will  result  in  added  costs,  and  prob¬ 
ably  poorer,  less  responsive,  overall  control. 

It  is  sometimes  feasible  to  negotiate  design  revision  locks  on  supplied  items.  This  scheme  does 
not  allow  the  supplier  to  change  the  product  without  notifying  the  customer,  and  the  customer  has  the 
option  of  not  accepting  a  proposed  change.  If  the  supplier  makes  changes,  the  revision  lock  agree¬ 
ment  ensures  that  spares  will  remain  available  to  the  old  design.  These  agreements  require  that  the 
customer  pay  a  fee  for  the  future  access  to  old  designs,  are  very  hard  to  negotiate  and  maintain  in  the 
real  world,  and  can  result  in  not  being  able  to  take  advantage  of  technology  updates  nor  cost  reduc¬ 
tions  in  newer  designs.  Therefore,  this  approach  should  only  be  used  for  the  support  of  products  that 
are  highly  mature,  where  the  market  technology  is  not  dynamic,  and  when  the  product  interface  to 
other  portions  of  the  system  has  a  high  number  of  second-order  specification  parameters  (characteris¬ 
tics  that  are  not  fully  defined  nor  controlled). 

Technology  infusion  plans  and  other  forms  of  pre-planned  (system)  product  improvements  need  to 
be  integrated  into  the  CM  plans.  This  will  allow  the  needed  interface  characterization  and  testing  to 
be  planned  and  adequately  funded  to  enable  the  system  to  be  managed  elfectively  throughout  the 
transition. 
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SYSTEM  LIFE-CYCLE  PLANNING 


System  life-cycle  planning  provides  long-term  plans  for  the  investments  needed  to  keep  addressing 
continuing  operational  requirements.  The  various  types  of  investments  range  from  modifying  sup¬ 
port  plans  to  major  technology  infusions  to  a  total  rearchitecture  of  the  system  partitioning  that 
amounts  to  a  system  replacement.  The  investment  alternatives  are  normally  considered  to  include 
these  five: 

•  Modifying  support. 

•  Product  improvements. 

•  Technology  infusions. 

•  System  upgrades. 

•  System  replacement. 

Most  system  requirements  address  operational  requirements  that  have  existed  for  a  considerable 
length  of  time  and  will  continue  to  exist  for  the  foreseeable  future.  For  instance,  there  has  been  a 
need  to  effectively  communicate  between  ships  as  long  as  there  have  been  navies.  However,  the 
employment  of  naval  battle  groups  has  changed  from  the  days  of  Roman  triremes  operating  locally 
to  modem  navies  having  aircraft  carriers  and  submarines  deployed  worldwide.  National  interests  are 
now  global  and  not  merely  the  immediate  coastline  or  fishing  areas.  As  a  result,  the  needs  to  com¬ 
municate  have  certainly  changed.  Of  course,  technology  has  changed  the  nature  of  naval  warfare 
even  in  the  structures  of  ships  and  weapons;  technology  has  also  changed  the  nature  of  supporting 
systems  such  as  communications  systems.  As  a  result,  a  system  designer  must  plan  for  a  relatively 
long  system  design  life,  even  though  the  products  composing  the  system  may  have  relatively  shorter 
life  spans  and  even  though  there  is  an  expectation  for  technological  improvements  that  will  be  incor¬ 
porated  into  the  system. 

There  are  nearly  always  major  differences  between  component  life,  product  life,  and  system  life. 
The  component  life  affects  reliability  and  support  planning  because  the  component  end-of-life  results 
in  a  component  failure  and  requires  a  maintenance  action.  The  product  life  is  determined  by  the  eco¬ 
nomic  support  period.  System  life  is  determined  by  the  viability  of  the  technical  approach  in  contin¬ 
uing  to  meet  operational  requirements.  In  any  system,  differences  in  component,  product,  and  sys¬ 
tem  life  spans  must  be  resolved.  As  the  practical  differences  in  product  life  and  system  life  may  be 
very  subtle,  there  tends  to  be  a  focus  on  effective  product  life  in  determining  when  to  initiate  various 
support  plans,  technology  infusions,  system  upgrades,  and  replacement  systems;  the  result  is  either  a 
failure  to  make  timely  investments  to  maintain  system  viability  or  to  invest  in  product  or  technology 
upgrades  that  are  untimely  and  ineffective.  ° 

At  least  eight  factors  are  considered  in  designing  the  system  life: 

•  Operational  requirements  stability. 

•  Operational  requirements  predictability. 

•  Ability  of  current  technology  to  address  the  current/predicted  operational  requirements. 

•  Predicted  technological  growth. 

•  Technological  cost  trends. 
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•  Potential  technological  sharing  between  markets. 

•  Existence  and  growth  of  technology  standards. 

•  Production  technology  trends. 

With  rare  exception,  the  operational  requirements  will  continue  well  beyond  the  product  life,  so  the 
system  designer  should  try  to  design  the  system  life  to  be  as  long  as  possible.  The  general  trends  of 
the  operational  requirements  evolution  will  often  be  sufficiently  predictable  to  allow  the  system 
designer  to  marry  appropriate  technologies  into  a  technical  approach  that  can  remain  viable  for 
decades.  Some  well-designed  systems  have  remained  viable  for  over  50  years,  although  30  years  is 
more  typical.  Current  technology  is  seldom  able  to  meet  the  anticipated  evolution  of  operational 
requirements;  in  fact,  current  technology  may  not  even  be  able  to  meet  all  of  the  current  operational 
requirements.  Therefore,  the  system  designer  must  select  the  best  available  technologies  and  create 
system  partitions  that  will  promote  the  cost-effective  future  exploitation  of  technology  growth.  At 
least  half  of  the  above  factors  are  intertwined  in  the  evolution  and  costs  of  the  technologies  that  may 
be  incorporated  at  some  future  time.  All  systems  will  be  evolved  over  time;  the  degree  of  system 
life-cycle  planning  will  determine  how  much  the  changes  will  cost  as  the  system  is  evolved.  These 
factors  must  be  considered  for  all  system  acquisitions  if  the  life-cycle  costs  are  to  be  adequately  con¬ 
trolled.  Good  system  life-cycle  planning  will  reduce  the  total  amount  of  documentation  needed, 
improve  overall  configuration  management,  maintain  high  levels  of  effectiveness  in  both  operations 
and  support,  and  provide  high  responsiveness  to  evolving  operational  requirements. 

System  support  modifications  only  change  the  items  supported  without  changing  the  system  archi¬ 
tecture.  Even  the  form,  fit,  and  function  specifications  for  the  product  level  do  not  change.  A  sup¬ 
port  modification  may  also  change  the  procedures  implemented  for  support  or  the  place  of  perfor¬ 
mance  of  support  actions  in  order  to  accommodate  changes  in  the  product  character  or  other  support 
circumstances.  A  new  piece  of  the  system  might  be  included  below  the  system  support  level  that 
may  require  prescreening  an  item  returned  for  repair  to  determine  which  depot  maintenance  activity 
should  take  the  action.  A  new  item  may  be  “the  same  except  for  repair  parts”  as  the  item  it  replaces. 

In  product  improvements,  the  system  architecture  remains  intact  and  unchanged,  but  the  product- 
level  implementation  is  changed  to  add  or  modify  functionality.  This  will  result  in  the  revision  of  the 
governing  form,  fit,  and  function  specifications  and  add  elements  to  interface  specifications.  Product 
improvements  can  be  preplanned  when  a  supporting  technology  is  not  sufficiently  mature  to  address 
the  required  functionality  when  a  system  is  fielded.  Preplanning  product  improvements  allows  the 
changes  to  be  inserted  by  engineering  change  proposal  rather  than  having  to  recourse  through  all  of 
the  acquisition  system  procedures  for  acceptance  and  test.  The  procedures  for  implementing  unpre¬ 
planned  product  improvements  are  still  simpler  than  initial  acceptance  of  a  new  system,  but  still 
require  some  rigorous  tests  and  reviews  of  effectiveness  and  suitability.  System  support  must  always 
be  modified  in  some  way  to  accommodate  the  changes.  An  extreme  example  is  to  design  a  plane  for 
a  more  powerful  engine  that  is  not  immediately  available  but  allowing  the  new  engine  to  be  installed 
as  soon  as  it  becomes  available  without  modifying  the  rest  of  the  plane. 

Technology  infusions  are  similar  to  product  improvements  and  are  treated  administratively  like 
product  improvements  in  the  acquisition  system  procedures.  However,  a  technology  infusion  goes 
beyond  the  scope  of  a  product  improvement  in  that  system  design  standards  above  the  product  level 
may  be  modified,  and  existing  product-level  system  items  may  require  modification  to  adapt  to  the 
new  interface  standards  imposed  by  the  new  technology.  The  most  cost-effective  approach  to 
technology  infusion  is  to  preplan  the  system  design  with  appropriate  interface  points  so  that  the  new 


57 


technology  can  be  integrated  without  modifying  any  system  components.  This  might  be  the  equiva¬ 
lent  to  changing  a  ship  from  steam  propulsion  to  gas  turbine  propulsion;  the  change  affects  the 
engines  but  also  requires  changing  the  ship  machinery  spaces  to  accommodate  the  new  engine  char¬ 
acteristics. 


A  system  upgrade  involves  rearchitecturing  a  major  portion  of  the  system  and  redocumenting  the 
interface  standards  to  accommodate  the  new  technology.  For  example,  changing  out  the  system 
board  in  a  personal  computer  to  upgrade  from  XT  technology  to  current  technology  would  involve 
not  only  changing  the  system  board  but  the  hard  drives,  disk  controller,  expansion  memory,  video 
card  (and  video  monitor  as  well  in  all  probability),  and  most  other  major  function  cards  because  the 
current  technology  uses  different  bus  structures  and  interface  standards.  Most  of  the  advantages  of 
the  new  technology  cannot  be  exploited  without  changing  the  interface  standards. 

When  none  of  the  above  alternatives  are  cost  effective,  the  system  is  replaced,  redesigning  the  sys¬ 
tem  from  the  top  down  with  new  partitioning  standards. 


Systems  employing  a  substantial  COTS  content  must  reconcile  the  relatively  short  product  life  of 
most  commercial  products  with  the  desired  long  system  life.  COTS  products  are  constantly  changed 
to  obtain  better  production  efficiencies  and  to  respond  to  market  requirements  that  may  have  nothin 
to  do  with  the  system  technical  requirements;  changes  of  this  nature  can  require  minor  adjustments  m 
the  system  support  implementation.  COTS  products  are  also  modified  and  evolved  to  incorporate 
new  technology  and  to  keep  a  competitive  edge  in  technologically  volatile  markets;  these  changes 
tend  to  be  very  substantial,  requiring  action  by  the  system  Life-Cycle  Management  Agent  (LCMA) 
(System  Design  Agent  or  In-Service  Engineering  Agent  (ISEA),  depending  on  the  system  phase). 
Most  of  these  evolved  products  are  changed  so  much  that  they  are  substantially  new  products  with 
very  little  in  common  with  the  prior  version.  A  typical  product  commercial  life  is  3  to  6  years, 
compared  to  a  good  system  life  (after  fielding)  of  30  to  40  years.  The  commercial  product  life  in 
some  high-technology  areas  is  under  1  year.  System  designers  strive  to  create  an  operationally  flex¬ 
ible  system  architecture  that  can  be  easily  adapted  to  new  operational  requirements,  but  substantial 
effort  is  still  delegated  to  an  ISEA  to  implement  planned  and  unplanned  changes.  The  primary  tools 
for  system  LCMAs  are  product  improvements  and  technology  infusions. 


Obviously,  a  product  life  changing  in  5  to  6  years  is  not  compatible  with  a  system  life  that  is  six 
times  as  long.  This  implies  that  some  major  system  restructuring  actions  will  have  to  be  done  by  the 
LCMA  during  the  system  life.  By  itself,  this  requires  the  system  acquisition  manager  to  plan  to  sup¬ 
port  the  LCMA  function  for  the  life  of  the  system  and  also  to  program  funding  to  allow  product 
improvement  actions  consistent  with  the  expected  product  life  spans.  These  can  be  major  factors  in 
computing  the  system  life-cycle  costs. 


It  is  a  recommended  best  practice  for  the  LCMA  to  establish  specification  control  drawings  for 
each  level  of  the  product  WBS  for  each  installation.  These  drawings  must  be  under  the  LCMA’s 
control  (rather  than  delegated  to  a  system  integration  contractor).  It  is  even  a  good  idea  to  key  the 
drawing  number  or  extensions  of  the  top-level  drawing  number  to  the  associated  work  breakdown 
numbers  (see  the  discussion  of  configuration  status  accounting  under  CONFIGURATION  MAN¬ 
AGEMENT  (CM)  ISSUES).  This  allows  support  planning  and  provisioning  actions  to  be  done 
against  these  stable  drawings  while  the  LCMA  performs  a  continuing  market  analysis  of  technology 
updates,  technology  infusions,  and  product  versions.  The  LCMA  can  then  perform  configuration 
status  accounting  against  the  products  actually  fielded  and  matrix  product  improvements  into  the 
field  by  reference  to  the  WBS-based  control  drawings.  Similarly,  technical  manuals  should  be  keyed 
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to  each  installation  and  to  the  product  work  breakdown.  This  is  especially  true  for  interactive  elec¬ 
tronic  technical  manuals.  In  this  way,  the  LCMA  can  maintain  consistency  between  the  product  data, 
support  data,  technical  manuals,  associated  training,  and  other  support  elements  in  spite  of  frequent 
product  updates  and  different  installation  configurations,  while  presenting  a  stable  support  environ¬ 
ment  to  the  user  community. 

Small  acquisition  projects  will  often  partition  their  systems  to  utilize  commonality  with  other  sys¬ 
tems  that  are  already  supported.  When  this  is  done,  it  is  essential  for  the  system  commonality  to  be 
managed  by  a  common  LCMA.  If  this  is  not  done,  no  economy  of  scale  can  be  achieved,  but  even 
worse,  the  systems  will  get  out  of  step  with  each  other  during  the  product  improvements  and  technol¬ 
ogy  upgrades  that  will  take  place  over  the  life  cycle.  This  usually  results  in  the  smaller  system(s) 
becoming  obsolescent  and  experiencing  higher  downtimes,  higher  costs,  and  lower  availabilities. 

The  common  LCMA  needs  to  ensure  that  all  systems  sharing  resources  share  common  upgrade  paths 
and  common  interface  specification  documentation  databases.  The  economies  that  can  be  achieved 
through  common  LCMA  support  can  amount  to  50  to  70  percent  under  the  costs  of  supporting  the 
systems  independently. 


SUPPORT  PLANNING 

Support  planning  for  NDI  acquisitions  is  similar  in  scope  to  planning  for  developmental  acquisi¬ 
tions,  except  that  the  constraints  are  somewhat  different.  For  the  various  off-the-shelf  alternatives, 
support  systems  are  already  in  place  that  may  or  may  not  be  suitable  for  the  intended  application. 
Support  planning  for  these  systems  needs  to  focus  on  what  elements  can  be  used  directly  versus  ele¬ 
ments  that  must  be  modified  or  developed.  For  the  modification  and  integrated  NDI  forms,  the  sup¬ 
port  planning  effort  is  normally  a  blend  of  adapting  existing  support  and  generating  the  support 
needed  for  the  new  or  modified  system  elements.  In  either  case,  NDI  acquisitions  proceed  much 
faster  than  the  time  it  takes  to  establish  final  support;  therefore,  all  NDI  acquisitions  become  highly 
dependent  on  interim  support,  especially  for  training  and  spares. 

NDI  acquisitions  are  often  plagued  by  inadequate  funding  for  support  elements.  This  is  a  result  of 
two  factors:  a  lack  of  adequate  system  planning  and  a  perception  that  the  planned  support  is  out  of 
scale  to  the  overall  acquisition.  The  first  arises  when  the  acquisition  managers  shift  into  a  purchas¬ 
ing  mode  rather  than  acknowledging  the  need  to  develop  the  system  requirements  against  which  the 
procurements  are  to  be  made.  The  second  occurs  when  acquisition  managers  fail  to  recognize  that 
the  support  planning  levels  are  relatively  constant  between  NDI  and  developmental  efforts,  but  the 
NDI  procurement  effort  is  significantly  less  than  a  development  effort.  This  means  that  life-cycle 
support  levels  of  effort  may  be  significantly  larger  portions  of  both  the  initial  acquisition  budget  and 
also  the  total  life-cycle  costs.  For  instance,  typical  developmental  acquisition  life-cycle  costs  break 
down  to  about  10  percent  research  and  development,  30  percent  production  and  installation,  and  60 
percent  life-cycle  support.  A  typical  NDI  acquisition  life-cycle  cost  will  break  down  to  about  3  per¬ 
cent  requirements  development  (the  equivalent  of  R&D),  17  percent  production  and  installation,  and 
80  percent  life-cycle  support.  In  a  typical  developmental  acquisition,  support  planning  normally 
constitutes  about  half  of  the  overall  R&D  effort.  In  NDI  acquisitions,  support  planning  is  about  75 
percent  of  the  requirements  development  effort.  These  differences  in  cost  distribution  often  deceive 
those  unfamiliar  with  NDI  and  can  lead  to  managers  underscoping  the  level  of  required  effort  in  the 
support  areas. 
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The  usual  tradeoffs  are  encountered  in  planning  support  between  contractor  support  and  organic 
support  or  some  mix  between  contractor  and  organic  support.  The  preference  is  toward  contractor 
support,  including  warranty  support  because  it  is  generally  the  most  cost-effective  support,  sharing 
support  resources  across  multiple  products.  However,  organic  support  generally  supplies  the  most 
responsive  support  with  the  lowest  system  downtimes.  Also,  organic  support  can  function  in  com¬ 
bat  environments  where  contractor  support  generally  is  not  available.  These  tradeoffs  are  influenced 
by  the  usage  and  technology  constraints.  For  instance,  contractor  support  is  more  responsive  to 
technology  changes  and  can  be  made  available  almost  immediately.  Organic  support  is  expensive  to 
adjust  for  technology  changes  and  can  take  years  to  establish  or  to  alter  to  accommodate  major 
changes.  Organic  support  is  especially  critical  when  high  operational  availability  is  required  for 
highly  mobile  or  deployed  systems.  Contractor  support  is  particularly  effective  for  fixed  site  sup¬ 
port,  especially  in  non-combat  environments.  It  is  usually  necessary  to  set  up  some  hybrid  support 
using  some  organic  maintenance  and  other  support  capabilities,  preferably  managed  by  a  central 
engineering  capability  such  as  the  ISEA,  backed  up  by  contractor  support. 

INTERIM  SUPPORT 

All  NDI  acquisitions  depend  heavily  on  interim  support.  Interim  support  is  provided  by  the 
acquiring  agency  rather  than  through  existing  support  agencies.  Costs  for  interim  support  must  be 
budgeted  by  the  acquisition  program  rather  than  being  absorbed  into  support  agency  operating  funds. 
Only  the  MILOTS,  GOTS,  and  some  FME  acquisitions  have  sufficient  existing  support  to  avoid  sig¬ 
nificant  interim  support  problems;  even  in  these  cases,  a  system  requiring  significant  increases  in 
support  levels  will  require  some  interim  support  while  the  existing  support  structure  adjusts  to  the 
new  demands.  Many  NDI  acquisitions  never  leave  the  interim  support  mode;  such  systems  are  usu¬ 
ally  subject  to  such  rapid  upgrades  that  final  support  could  never  be  established  quickly  enough  to 
justify  a  transition  from  interim  support.  Other  NDI  acquisitions  are  low-population  systems  that  are 
more  effectively  supported  through  interim  support  measures.  Virtually  all  NDI  acquisitions  are  so 
rapid  that  interim  support  is  required;  in  fact,  some  acquisitions  are  so  fast  that  it  is  a  challenge  to 
even  establish  interim  support  fast  enough  to  field  a  supported  system.  In  any  case,  interim  support 
planning  and  timely  execution  are  critical  to  the  success  of  NDI  acquisitions. 

Intenm  support  depends  heavily  on  good  engineering  expertise  to  mediate  support  decisions; 
therefore,  it  is  important  to  identify  and  establish  the  ISEA,  Software  Support  Activity  (SSA),  and 
System  Training  Agent  (STA)  functions  early.  Interim  support  will  normally  use  contractor  support 
(warranty  or  field  engineering)  overseen  by  the  engineering  expertise  representing  the  acquisition 
agency.  Ideally,  the  ISEA,  SSA,  and  STA  should  each  participate  in  the  generation  of  procurement 
documents  and  in  the  source  selection  process.  This  allows  each  activity  to  gain  valuable  informa¬ 
tion  needed  to  establish  and  maintain  interim  support  very  efficiently,  to  identify  the  issues  that  will 
drive  support,  and  to  assign  personnel  with  appropriate  skills.  Alternatively,  the  system  design  activ¬ 
ity  may  be  charged  with  the  responsibility  of  planning  and  executing  interim  support  if  the  system 
design  activity  has  the  appropriate  expertise.  In  either  case,  it  is  desired  that  the  interim  support 
planners  conduct  a  top-level  logistic  support  analysis  during  the  system  design  phase  while  partition¬ 
ing  decisions  are  being  made.  This  allows  the  support  considerations  to  be  effectively  integrated  into 
the  system  partitioning  decisions. 

Virtually  all  systems  incorporating  NDI  have  architectures  that  require  a  close  coordination 
between  the  ISEA,  SSA,  and  STA  functions  (a  fact  justifying  the  establishment  of  a  LCMA).  These 
architectures  have  functional  interfaces  driven  by  existing  “packaging”  of  functionality  rather  than 
by  operator-observable  functions.  Problems  are  likely  to  cross  system  interface  boundaries  and  to  be 
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more  difficult  to  accurately  diagnose  and  resolve.  The  combined  ISEA/SSA/STA  expertise  is  often 
required  to  develop  a  true  fix  to  the  problem.  For  instance,  a  communication  system  having  some 
NDI  components  was  found  to  have  a  high  number  of  software  trouble  reports  that  could  not  be 
duplicated  in  the  test  verification  system;  it  was  found  that  an  error  in  the  system  technical  manuals, 
which  were  also  the  basis  of  training,  had  an  error  resulting  from  a  change  in  an  NDI  component  that 
made  it  appear  that  there  was  a  software  problem.  The  SSA  had  insufficient  information  to  recog¬ 
nize  the  source  of  the  problem,  and  the  ISEA  and  STA  were  not  communicating  with  the  SSA.  The 
problem  was  finally  recognized  by  an  engineer  from  the  original  system  design  agency  who  hap¬ 
pened  to  be  aboard  a  ship  during  an  exercise  when  the  problem  showed  up.  The  engineer  recognized 
the  nature  of  the  problem  and  notified  all  of  the  agents  to  correct  the  error.  Problems  of  this  nature 
can  be  common  in  systems  architectures  using  NDI,  especially  if  the  system  has  a  high  number  of 
technological  upgrades  being  done  over  a  short  period  of  time  or  many  different  fielded  configura¬ 
tions. 

Interim  suppon  planning  must  provide  for  spare  parts,  hardware  maintenance,  software  support, 
training,  and  technical  documentation  maintenance.  Configuration  management  is  essential  for  car¬ 
rying  these  tasks  out  effectively.  Each  of  these  areas  have  their  own  unique  problems  that  are  dis¬ 
cussed  in  separate  sections  of  this  document.  In  order  to  accomplish  the  tasks  effectively,  the  System 
Design  Agent  (SD.Ai  or  LCMA  should  be  tasked  (and  funded)  to  bring  the  ISEA,  SSA,  and  STA 
functions  on  line  as  soon  as  possible  to  participate  in  the  acquisition.  Also,  the  SDA  should  be 
tasked  to  perform  a  top-level  logistics  support  analysis  integral  to  the  system  design  and  with  the  par¬ 
ticipation  of  the  support  agents.  This  will  allow  the  issues  arising  from  personnel,  human  factors, 
quality,  reliabilit>.  configuration  management,  documentation,  training,  maintenance,  and  other  ILS 
areas  to  be  effecti\el\  considered  and  integrated  into  the  system  support  plans.  It  will  also  allow 
funding  for  interim  support  to  be  identified  early  enough  to  be  put  into  the  budget  in  time  for  the  sup¬ 
port  to  be  on  line  u  hen  the  system  is  ready  for  fielding. 

Special  problems  u  ill  occur  when  dealing  with  the  rapid  integration  of  NDI.  Rapid  integration 
projects  can  design  and  field  a  system  capability  within  6  months  or  less.  In  such  programs,  system 
design,  NDI  procurement,  installation,  generation  of  technical  documentation,  and  support  planning 
must  proceed  in  parallel.  The  execution  of  all  of  these  tasks  must  be  well  coordinated  and  flexible 
enough  to  accommodate  changes  that  will  inevitably  occur.  Weekly,  if  not  daily,  coordination 
between  task  leaders  is  mandatory.  Most  rapid  integration  projects  experience  difficulties  because 
the  product  selection  and  integration  must  precede  technical  documentation,  test  planning,  and  train¬ 
ing  planning.  lea\  ing  insufficient  calendar  time  to  accomplish  these  functions  in  traditional  ways. 

The  system  integration  tests,  installations  and  checkouts,  and  initial  training  must  all  be  very  well 
coordinated  since  the\  will  all  be  done  within  approximately  the  same  brief  calendar  window.  Sys¬ 
tem  documentation  will  often  be  delayed  until  after  initial  fielding,  or  it  will  be  available  in  very  pre¬ 
liminary  forms.  To  meet  these  challenges,  the  following  steps  are  strongly  recommended; 

•  Identify  and  incorporate  the  different  types  of  expertise,  especially  support  system  expertise, 
that  is  needed  to  acquire,  field,  and  support  the  system  and  include  the  expertise  as  an  integral 
and  interactive  part  of  the  system  acquisition  team. 

•  Establish  a  strong  developmental  configuration  baseline  coordinated  by  the  system  designer  or 
integrator. 

•  Ensure  close  coordination  between  all  task  leaders,  with  at  least  weekly  meetings  to  discuss  all 
issues  from  each  perspective. 
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Plan  for  the  manpower/time  for  the  system  designers  to  develop  the  technical  documentation 
needed  at  the  system  level. 

Coordinate  system  integration  testing,  installation  and  checkout  activities,  and  system  training. 
It  is  often  possible  for  these  activities  to  share  facilities  and  personnel  to  the  mutual  benefit  of 
all  tasks. 

Design  system  training  to  also  provide  sufficient  system  overview  information  to  allow  for  late 
delivery  of  system  technical  documentation.  The  resulting  information  must  be  in  a  form 
accessible  to  the  trainees  after  the  system  is  fielded  and  operational. 

•  Procure  system  spares  with  the  initial  buy. 

•  Plan  for  one  or  more  complete  systems  to  remain  with  the  LCMA.  This  will  allow  verification 

testing  for  any  future  system  changes  and  also  provide  a  platform  for  emulating  and  diagnos¬ 
ing  field  problem  reports.  ° 

•  Provide  for  post-fielding  on-site  support  for  both  maintenance  and  training  for  at  least  6 
months  (calendar  time)  per  site. 

•  Establish  support  coordination  through  the  LCMA,  including  the  establishment  of  ISEA,  SSA, 
and  STA  functions.  Make  the  SDA  the  single  focal  point  for  all  required  field  support.  This 
will  allow  support  issues  to  be  effectively  identified  and  resolved  while  coordinating  future 
installations  and  system  design  changes. 

•  Anticipate  that  changes  will  be  dictated  by  issues  that  emerge  after  the  initial  fielding  of  the 
system.  These  changes  should  also  include  actions  that  will  make  the  system  more  support¬ 
able. 

Rapid  integration  projects  require  many  decisions  to  be  made  “off-the-cuff’  by  the  best  expertise 
available.  If  good  expertise  has  been  brought  to  bear  on  each  issue,  good  decisions  will  be  made; 
however,  some  corrections  should  also  be  expected.  A  rapid  integration  program  must  plan  and  fund 
follow-on  support.  The  scope  of  this  follow-on  support  may  equal  or  exceed  the  initial  acquisition. 
The  follow-on  support  must  then  transition  to  interim  support. 

WARRANTY  SUPPORT 

Warranties  are  expressions  of  the  supplier’s  confidence  that  a  product  will  meet  or  exceed  the  spe¬ 
cified  requirements  for  a  stated  period  of  time  under  specified  usage  conditions.  The  warranty  is 
generally  beneficial  to  both  the  supplier  and  the  customer.  The  customer  gets  assurance  that  the 
needs  satisfied  by  the  product  will  be  fulfilled  or  the  condition  will  be  remedied.  The  supplier  can 
generally  charge  a  little  more  (in  the  warranty  contingency)  than  would  otherwise  be  possible  and 
still  enjoy  high  customer  satisfaction.  If  a  product  is  particularly  good,  the  warranty  contingency 
becomes  extra  profit  for  the  supplier.  If  the  product  is  a  “lemon,”  the  customer  still  obtains  the 
required  service  at  no  added  charge,  no  matter  how  extensive  that  service  might  be.  Therefore,  sup¬ 
pliers  have  a  market-driven  incentive  to  build  very  good  products  covered  by  very  good  warranties. 

Every  product  is  supplied  with  a  warranty,  either  expressed  or  implied.  Virtually  all  commercial 
products  are  provided  with  a  standard  expressed  warranty  that  is  issued  to  limit  the  application  of  an 
implied  warranty  and  to  make  warranty  services  a  fixed  cost  for  the  product.  The  standard  warranty 
is  normally  relatively  short  in  term  (30  days  to  1  year)  and  is  intended  to  cover  latent  defects  and 
infant  mortality  failures.  Extended  warranties  are  often  options  that  allow  the  customer  to  fix  future 
costs  over  a  major  portion  of  the  intended  usage  life  of  the  product.  Some  markets  offer  lifetime 
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warranties,  either  because  the  product  is  of  such  quality/reliability  that  it  is  unlikely  to  require  service 
within  a  user’s  life,  because  the  product  is  associated  with  another  product  that  has  a  definable  lim¬ 
ited  life,  or  because  the  company  is  trying  to  protect  a  market  niche  from  its  competitors. 

Most  commercial  warranties  are  of  little  benefit  to  military  applications  for  one  or  more  of  the  fol¬ 
lowing  reasons; 

•  The  warranty  term  is  too  short.  (The  item  may  not  even  be  put  into  service  before  the  war¬ 
ranty  expires.) 

•  The  warranty  is  unenforceable  because  of  field  application  conditions,  environmental 
extremes,  organizational  maintenance  requirements,  or  actions  of  field  personnel  violating 
warranty  provisions. 

•  The  warranty  service  provisions  are  not  sufficiently  responsive  to  maintain  the  required  opera¬ 
tional  availability. 

•  The  warranty  is  too  limited  in  scope. 

•  The  warranty  requires  actions  that  are  impractical  to  implement  (such  as  a  quarterly  inspection 
by  an  authorized  representative). 

•  The  warranty  only  applies  to  the  acquiring  activity  and  cannot  be  transferred  to  the  user 
activity. 

Most  NDI  acquisitions  should  use  negotiated  warranties  tailored  to  the  acquisition  requirements. 
Negotiated  warranties  provide  the  benefits  of  commercial  warranties  while  removing  the  limitations 
described  above. 

NDI  acquisitions  normally  will  negotiate  warranties  that  are  long  term  and  with  enforcement  crite¬ 
ria  tailored  to  the  organizational  use  and  maintenance  environment.  The  following  steps  are 
necessary  to  establish  a  useful  negotiated  warranty  clause  that  can  be  realistically  bid  by  the  potential 
supplier  and  mutually  enforced: 

1 .  The  term  should  be  established  to  be  comparable  to  the  minimum  expected  product  life  within 
the  system.  (This  is  normally  about  5  years.)  The  term  should  be  specified  in  operating  or 
usage  hours  rather  than  calendar  days,  if  at  all  possible.  For  instance,  a  system  expected  to  be 
used  3000  hours  per  year  could  have  a  warranty  term  of  15,000  hours  (the  5-year  expected  life 
times  3000  hours  per  year).  To  make  this  provision  enforceable,  a  meter  or  other  measuring 
device  must  be  provided  to  keep  a  record  of  the  usage;  otherwise,  some  accepted  measure 
(such  as  steaming  hours  or  a  multiplier  of  steaming  hours,  a  value  that  can  be  retrieved  from 
the  Navy  maintenance  record  databases)  should  be  negotiated.  If  usage  hours  are  not  used,  the 
calendar  time  of  the  warranty  should  start  from  the  completion  of  installation  and  checkout. 
Also,  special  marking  provisions  are  normally  required  for  warranted  items  to  prevent  unau¬ 
thorized  or  inadvertent  maintenance  actions  that  would  violate  the  warranty. 

2.  The  operator  and  maintenance  skill  requirements  must  be  specified  and  consistent  with  the 
actual  field  personnel.  The  supplier  must  be  responsible  for  ensuring  that  support/test  equip¬ 
ment,  manuals,  training,  human  factors,  test  features,  and  all  other  factors  affecting  operator/ 
maintainer  interaction  with  the  product  is  adequate  for  the  specified  performance.  If  there  are 
excess  tum-ins  due  to  poor  documentation  or  training,  the  contractor  should  be  responsible  for 
bearing  the  costs  and  correcting  the  problems.  The  acquiring  activity  must  ensure  that  all 
operator  and  maintenance  personnel  using  or  maintaining  the  product  meet  the  specified  skill/ 
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experience  levels  and  are  available  to  the  contractor  or  the  contractor’s  agent  for  training. 
When  the  system  is  an  integration  of  warranted  items,  the  warranty  provisions  must  recognize 
an  adjudication  of  the  reported  problems  by  the  acquiring  activity  agent  (typically  the  ISEA). 

3.  The  ISEA  function  must  be  established  to  manage  the  warranted  items.  The  management 
function  includes  maintaining  warranty  records  for  each  product,  screening  items  returned  for 
service  for  false  removal,  coordinating  the  shipping  of  items  between  the  field  and  the  contrac¬ 
tor,  and  ensuring  the  warranty  provisions  are  being  met.  (The  ISEA  does  not  necessarily 
accomplish  all  of  these  tasks,  but  merely  directs  their  accomplishment.  The  supplier  or  a  third 
party  may  be  tasked  to  accomplish  some  of  these  tasks.)  The  warranty  enforcement  usually 
requires  reporting  at  the  field  or  organizational  level,  but  this  should  be  constructed  so  that  no 
new  or  unusual  reporting  procedures  are  imposed.  This  requires  the  warranty  manager  to 
extract  the  warranty  reports  from  existing  maintenance  record  systems. 

4.  The  supplier  should  be  tasked  to  perform  installation  and  checkout  or  to  observe  installation 
and  checkout  of  each  system  to  ensure  that  the  warranted  items  are  properly  installed  and  func¬ 
tioning  at  the  initiation  of  the  warranty.  If  the  warranted  item  is  to  be  modified  or  integrated 
into  a  larger  system  entity,  the  supplier  should  be  tasked  to  certify  the  modification  or  fntegra- 
tion  design  as  being  suitable  and  within  the  scope  of  the  intended  use  of  the  item. 

5.  Appropriate  quality/warranty  clauses  should  be  included  in  the  contract  and  tailored  to  the  rea¬ 
sonable  needs  of  the  acquisition.  The  contractor  should  be  liable  for  all  costs  for  the  products 
failure  to  perform  to  the  specified  requirements.  This  should  include  compensation  for  added 
costs  if  the  product  has  excess  failures  (beyond  those  allowed  bv  the  specified  reliability V 
however,  these  added  costs  must  be  identified  and  reasonable.  Failures  due  to  combat  should 
be  explicitls  excluded.  There  are  several  clauses  (under  Quality  Clauses,  Section  52)  available 
for  tailoring  m  ihe  Federal  Acquisition  Regulations  that  should  be  selected  on  the  basis  of  the 
type  of  NDl  being  warranted.  Specifically,  the  clause  entitled  “WARRANTY  OF  SYSTEMS 
AND  EQL  IIWIENTS  UNDER  PERFORMANCE  SPECIFICATIONS  OR  DESIGN  CRI¬ 
TERIA"  is  particularly  useful. 

6.  The  contractor  must  be  free  to  improve  the  product  and  to  provide  upgrades.  The  ISEA  must 
assure  that  an\  upgrades/improvements  still  conform  to  system  specifications. 

7.  The  warrant)  should  not  be  tied  to  a  specific  platform  installation  or  configuration.  (Some 
commercial  u  arranties  do  not  allow  movement  of  equipment  or  software  from  one  site  to 
another.)  The  gm  emment  should  retain  the  right  to  establish  pipeline  spare  pools  to  improve 
field  availabilitv  or  to  change  installation  sites  or  configurations  within  established  bounds.  It 
may  be  necessarx  to  change  the  mode  of  support  from  a  two-tier  to  a  three-tier  mode  in  order 
to  achieve  the  required  system  availability. 

The  negotiated  warranty  clause  should  normally  be  a  separately  priced  option.  It  is  important  to 
expose  the  warranty  cost,  and  the  warranty  cost  should  be  reasonable.  If  contractor  support  services 
of  similar  scope  are  also  bid  as  a  separately  priced  option,  the  warranty  cost  should  be  slightly  higher 
(by  5  to  10  percent)  than  the  contractor  support  cost.  This  added  cost  is  a  measure  of  the  risk  being 
assumed  by  the  contractor  at  fixed  cost.  A  fixed  warranty  cost  in  this  range  should  be  considered  ° 
equivalent  in  value  to  contractor  support  cost  at  the  lower  price  since  the  contractor  support  cost  will 
be  adjusted  in  rate  each  year,  and  the  demand  for  service  will  tend  to  rise  near  the  end  of  the  product 
life.  In  addition,  the  warranty  provides  an  incentive  for  the  contractor  to  improve  the  fielded  product 
reliability  since  the  unused  warranty  cost  becomes  added  “profit”  to  the  contractor.  Use  of  the  nego¬ 
tiated  warranty  clause  is  especially  important  for  Integrated  NDI. 
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CONTRACTOR  FIELD  MAINTENANCE  AND  OTHER  CONTRACTOR  SUPPORT 

Virtually  all  COTS  suppliers  maintain  some  form  of  customer  support.  The  level  of  support  varies 
between  suppliers  and  markets,  but  is  generally  stable  for  common  products  within  a  market.  The 
range  of  field/customer  support  capabilities  includes  the  following: 

•  Customer  support  line  telephone  service. 

•  Online  BBS  or  Web  Page  support. 

•  Toll-free,  24-hour  customer  support  line. 

•  Worldwide  customer  support  line. 

•  Worldwide,  toll-free,  24-hour  customer  support  line. 

•  Factory  service. 

•  Prorated  product  replacement. 

•  Total  product  replacement/exchange. 

•  Field  office  service. 

•  Worldwide  field  office  service. 

•  On-site  service  (domestic). 

•  On-site  service  (worldwide). 

•  Combinations  of  the  above. 

•  Contractor  facility  training. 

•  On-site  training. 

•  Contract  course  training  support. 

The  extensiveness  of  the  service  is  influenced  by  the  size  of  the  company,  the  breadth  of  the  product 
line  (allowing  products  to  share  the  common  support),  product  support  demands  (failure  rates  and 
support  complexity),  and  the  market  requirements.  Each  of  these  support  capabilities  provides  a 
degree  of  responsiveness  and  a  region  of  coverage,  but  at  a  cost.  The  range  of  capabilities  might  be 
an  explicit  customer  option  or  included  in  the  product  price. 

WARRANTY  SERVICE  VERSUS  CONTRACTOR  SUPPORT 

The  degree  of  field  support  is  often  a  negotiable  item  for  major  NDI  acquisitions.  It  is  useful  to 
request  quotes  for  desired  levels  of  contractor  support  as  contract  options.  The  quotes  can  be  particu¬ 
larly  useful  when  they  are  compared  to  similar  quotes  for  warranty  services.  If  there  is  a  very  signifi¬ 
cant  difference  between  the  contractor  service  quote  and  the  warranty  service  quote,  it  indicates  that 
the  supplier  is  not  confident  of  the  product  support  demands  (if  the  warranty  is  bid  higher)  or  that  the 
level  of  support  requested  is  beyond  the  supplier’s  normal  support  capabilities  (if  the  warranty  is  bid 
lower).  For  similar  levels  of  support,  the  warranty  should  normally  be  bid  at  less  than  10  percent 
over  contractor  service,  with  less  than  5  percent  more  typical  for  mature  products.  The  difference 
between  the  warranty  service  quote  and  the  contractor  service  quote  is  called  the  “warranty  pre¬ 
mium”  or  the  “supplier  risk  cost.”  Past  experience,  together  with  life-cycle  cost  analyses  of  future 
cost  implications,  shows  that  a  warranty  premium  not  exceeding  four  times  the  assumed  average  rate 
of  inflation  is  equal  in  real  value  to  contractor  service  agreements  with  annual  rate  adjustments. 
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SPARE  PARTS 


The  level  of  sparing  is  impacted  by  availability  requirements,  reliability  performance,  and  costs.  It 
would  be  nice  to  never  need  spare  parts,  and  if  nothing  ever  breaks,  spare  parts  are  never  demanded. 
However,  even  highly  reliable  systems  require  spare  parts  to  cover  those  contingencies  of  unforeseen 
circumstances  so  that  the  system  can  be  maintained  and  be  available  for  use.  Vital  systems  and  mis¬ 
sion-critical  equipment  have  availability  requirements  that  cannot  tolerate  long  downtimes  awaiting 
parts,  so  spares  for  such  items  must  be  positioned  for  ready  accessibility  if  they  are  demanded.  On° 
the  other  hand,  every  replaceable  item  cannot  be  spared  at  the  organizational  level,  even  for  the  most 
vital  or  mission-critical  equipment  because  of  cost  and  space  constraints  for  deployed  units.  (In 
the  1970s,  the  Naval  Material  Command  estimated  that  an  aircraft  carrier  would  need  to  be  120  per¬ 
cent  larger  in  order  to  have  sufficient  storage  space  for  all  of  the  spare  parts  needed  to  support  only 
designated  vital  and  mission-critical  items;  the  cost  of  the  spares  was  estimated  to  be  equal  to  the 
original  cost  of  the  ship.  Similarly,  a  fleet  ballistic  missile  submarine  would  have  had  to  displace 
over  100,000  tons  to  maintain  every  possible  spare  on  board  throughout  its  deployment.)  System 
planners  must  take  these  logistic  constraints  and  support  risks  into  account,  while  designin'^  for  max¬ 
imum  operational  availability.  ° 

It  is  very  desirable  to  keep  the  number  of  new  items  of  supply  to  an  absolute  minimum.  When 
fewer  new  items  of  supply  are  needed,  there  are  more  options  for  providing  depth  of  support  cost 
effectively  and  maintaining  a  very  high  operational  availability.  In  development  systems,  standard¬ 
ization  programs  can  be  used  to  reduce  the  new  logistics  items  that  need  to  be  supported  with  spares; 
however,  this  is  usually  not  practical  for  NDI.  There  may  be  some  flexibility  in  the  system  partition¬ 
ing  to  maximize  the  standards  within  the  system,  thereby  reducing  the  number  of  new  items  to  be 
introduced,  but  this  is  made  significantly  more  difficult  by  many  of  the  new  technologies  and  rapidly 
evolving  technologies.  In  some  cases,  some  redesign  or  modifications  can  reduce  the  spare  part 
requirements.  For  instance,  one  system  had  48  “computer-on-a-card”  items  in  a  common  chassis 
consisting  of  nine  different  card  types  but  enabling  the  use  of  off-the-shelf  software.  By  modifying 
the  software  to  operate  on  a  single  “computer-on-a-card”  type,  it  became  feasible  to  provide  a  spare 
card  in  the  same  chassis  to  guarantee  spare  availability.  The  minor  software  modifications  allowed  a 
significant  improvement  in  system  availability  (from  0.45  to  0.987)  while  reducing  overall  life-cycle 
costs  very  dramatically  simply  by  reducing  the  overall  spare  part  requirements.  A  further  reduction 
m  hfe-cycle  costs  was  achieved  by  selecting  a  card  type  of  the  original  nine  that  was  already  provi¬ 
sioned  for  other  systems.  These  two  actions  resulted  in  an  overall  reduction  of  spare  part  costs  by 
93  percent.  To  achieve  these  savings,  good  standards  had  to  be  selected  for  the  system  partitionin<^ 
based  on  both  industry  market  standards  and  the  standards  adopted  by  the  military  market  (resultincr 
in  systems  sharing  the  supply  base). 

Both  reliability  (failure  rate)  data  and  environmental  performance  data  are  essential  to  makin<^ 
informed  sparing  decisions.  Much  of  the  required  information  can  be  obtained  during  the  market 
survey  and  screening  processes.  However,  the  confidence  factors,  as  discussed  under  RELIABIL¬ 
ITY  ISSUES,  must  be  taken  into  account.  Information  of  low  confidence  levels  will  lead  to  over¬ 
sparing  and  unnecessary  expense  or  undersparing  with  poor  operational  availability  performance. 

Each  dollar  spent  in  developing  higher  confidence  information  through  .screening  tests  can  re.tnm  in 
excess  of  one,  hundred  dollars  in  reduced  spare  part  costs.  Tests  are  considered  complete  when  the 
confidence  levels  have  been  improved  (when  combined  with  all  available  information)  so  that  the 
predicted  insurance  spares  have  been  reduced  to  less  than  10  percent  of  the  total  predicted  spares  (or 
to  one  spare).  Technical  Document  108  provides  additional  information  on  Support  Parameters  and 
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the  practical  constraints  for  different  application  environments  that  should  be  considered  in  the  Con¬ 
ceptual  Phase. 

The  difference  between  system  life  and  product  life  must  also  be  taken  in  account  when  making 
sparing  decisions.  If  spares  are  provided  for  the  system  life,  most  of  the  spares  will  never  be  needed 
because  the  spared  item  will  be  replaced.  The  practical  life  of  high-reliability  products  will  probably 
not  exceed  the  capacity  of  insurance  spares  that  are  provided  merely  to  ensure  availability.  The 
spares  for  commercial  items  should  be  planned  for  the  life  of  the  product  in  the  system.  In  most 
cases,  the  system  life-cycle  planning  will  indicate  system  upgrades  amounting  to  about  10  percent  of 
the  total  system  functionality  per  year  as  a  minimum  before  product  life  factors  are  taken  into 
account.  However,  if  the  average  life  of  a  commercial  product  is  5  years  and  commercial  support  for 
the  product  is  guaranteed  for  only  5  years  beyond  the  product  phase-out,  system  upgrades  amounting 
to  as  high  as  20  percent  of  system  functionality  per  year  might  be  required.  Most  commonly,  the 
product  will  be  incorporated  into  the  system  after  it  has  been  on  the  market  2  to  3  years,  so  the  sys¬ 
tem  is  fielded  with  only  2  years  of  product  life  remaining  (typical).  The  commercial  spares  availabil¬ 
ity  can  then  be  expected  for  only  another  7  years.  However,  each  product  must  be  evaluated  on  a 
case-by-case  basis  for  both  product  maturity  and  supplier  support.  Spares  are  only  required  for  the 
period  of  time  from  the  fielding  of  the  system  to  the  replacement  of  the  product  by  a  system  upgrade. 
Ideally,  each  spare  will  run  out  at  the  same  time  that  the  last  spared  item  is  removed  from  the  system. 
In  practice,  the  insurance  spares  should  be  left  over  and  need  to  be  excessed,  since  they  represent  the 
number  of  spares  needed  to  reduce  availability  risks  to  acceptable  levels.  The  time  span  required  for 
the  insurance  spares  needs  to  span  the  remaining  product  life  plus  the  commercial  availability  of 
spares  time  up  to  the  planned  system  upgrade  period  plus  2  years.  The  2-year  added  time  results 
because  funding  for  system  upgrades  cannot  be  assured,  especially  when  the  budget  is  being  drawn 
down;  therefore,  an  added  budget  cycle  (2  years)  is  required  to  reduce  budget  risk  to  the  planned  sys¬ 
tem  upgrade. 

SPARING  STRATEGIES 

Spares  may  be  provided  at  any  or  all  of  up  to  four  echelons  of  maintenance  support,  depending  on 
the  support  strategy  and  using  agency  policies  and  required  operational  availability.  Organic  spares 
provide  the  highest  operational  availability,  but  are  limited  by  constraints  of  space  and  cost.  Techni¬ 
cal  Document  108  provides  additional  information  on  Support  Parameters  to  be  considered  in  the 
Conceptual  Phase  and  the  constraints  of  various  organizational  levels  of  support  capability.  Spares 
maintained  by  an  original  equipment  manufacturer  (OEM)  or  supplier  are  least  costly  since  they  are 
only  purchased  on  an  as-needed  basis;  however,  the  lead  times  for  the  spare  availability  can  be 
months  under  some  circumstances — not  acceptable  for  good  operational  availability.  All  sparing 
strategies  try  to  balance  achieving  minimum  costs  and  maximum  availability  within  the  system 
constraints. 

Embedded  spares  and  spares  kits  are  good  approaches  for  vital  items  with  high  operational  avail¬ 
ability  requirements  for  COTS,  ROTS,  FOOTS,  FME,  most  modified  NDI,  and  Integrated  NDI. 

These  types  of  NDI  tend  to  have  high-value  replacement  items  that  are  not  military  standard  stock 
items.  Nevertheless,  the  supply  parts  control  centers  or  national  stock  system  should  be  queried  to 
determine  if  items  being  considered  for  sparing  are  already  provisioned  in  another  system.  This 
information  will  also  appear  in  a  good  item  screen  conducted  during  the  system  partitioning  phase. 
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To  minimize  expensive  organic  spares,  supplemental  spares  can  be  provided  at  an  intermediate  or 
depot  level.  The  following  alternatives  for  depot-level  sparing  should  be  considered; 

•  OEM  support. 

•  Prime  system  integrator  support. 

•  Contractor  field  service  support. 

In-Service  Engineering  Agent  (ISEA)  controlled  support  with  contractor  back-up  support. 

•  Supply  system  stocking. 

•  Life-of-Type  procurement  stocking  (where  the  designated  spares  are  maintained  outside  of  the 
military  supply  system,  usually  under  the  direction  or  control  of  the  ISEA). 

•  Escrow  data  rights  (where  data  rights  are  purchased  at  a  reduced  cost  and  “held  in  escrow”  for 
the  contingency  that  if  the  supplier  may  goes  out  of  business  or  ceases  to  support  the  item,  a 
third  party  can  use  the  data  to  produce  the  item). 

Purchased  data  rights  (where  the  complete  design  disclosure  is  purchased  from  the  original 
supplier  for  purposes  of  obtaining  additional  sources  of  supply  in  the  future). 

High-value  items  are  normally  controlled  through  the  ISEA  in  order  to  avoid  a  high  false  removal 
rate.  High  false  removal  rates  artificially  inflate  pipeline  spare  requirements  and  usually  lead  to  con¬ 
tract  disputes  over  warranty  clauses.  Also,  major  warranted  items  should  be  controlled  through  the 
ISEA.  Lower  value  items  and  minor  warranted  items  can  use  direct  contractor  support,  although  a 
common  process  for  handling  all  items  warranted  by  a  single  supplier  is  desired  (even  for  different 
system  applications).  Life-of-Type  procurement  should  only  be  used  for  unique  items  (such  as 
application  specific  integrated  circuits  or  heavily  customized  modules)  that  will  not  be  maintained  in 
manufacture  or  for  support  bridging  between  the  end  of  contractor  support  and  product  phase-out. 
Data  rights  should  only  be  purchased  for  vital  and  unique  system  elements  where  there  is  a  signifi- 
of  fhe  loss  of  the  original  source  of  supply  and  where  the  technology  is  anticipated  to  be 
transferable  to  another  source.  The  responsiveness  of  contractor  support  access  through  the  supply 
system  or  through  an  ISEA  can  be  improved  by  several  techniques.  One  such  technique  is  to  estab¬ 
lish  blanket  ordering  agreements  with  the  supplier  that  allow  telephone  or  facsimile  orders  to  be 
placed.  Another  is  called  the  Just-In-Time  Paperless  Ordering  Procurement  System  (JIT-POPS); 
JIT-POPS  places  orders  electronically  (effectively  an  e-mail  procurement),  even  allowing  direct 
messaging  from  the  user  organization.  Contractor  field  service  support  can  be  expedited  through  a 
special  prearranged  “credit  card”  agreement.  Suppliers  can  (for  a  small  fee)  be  required  to  maintain 
a  guaranteed  stock  to  support  any  of  the  rapid  ordering  methods.  Any  of  the  on-demand  techniques 
have  the  advantage  that  expensive  pipeline  spares  and  other  inventory  spares  are  either  reduced  or 
eliminated. 

MILOTS  and  GOTS  items  normally  have  an  established  sparing  strategy  already  implemented. 
However,  the  existing  strategy  may  not  be  appropriate  for  the  particular  application.  For  instance,  a 
system  previously  provisioned  for  use  on  an  aircraft  carrier  will  require  a  different  strategy  if  it  is 
used  by  special  forces  teams.  Wherever  possible,  an  existing  strategy  should  be  used. 

The  methods  employed  to  support  NDI  systems  should  not  require  modification  of  either  the  sup¬ 
plier  s  support  infrastructure  or  the  military  supply  procedures  or  maintenance  reporting  procedures 
A  well-designed  sparing  system  blends  the  strengths  of  both  support  systems  and  ensures  that  timely 
and  accurate  information  is  obtained  to  make  cost-effective  repair/replace/upgrade  decisions. 
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DYNAMIC  SPARES 


Many  COTS  products  are  changed  so  frequently  that  every  time  an  order  is  placed,  the  part  is 
slightly  different.  Many  times  the  differences  are  of  no  concern  because  the  form,  fit,  and  function 
of  the  original  item  are  maintained;  however,  changes  outside  of  this  realm  can  be  of  major  concern 
to  the  user  and  support  agents  alike.  Products  in  this  category  should  normally  be  spared  through  the 
ISEA,  and  a  special  screening  function  should  be  implemented  under  the  ISEA  direction  to  ensure 
that  incoming  spares  are  compatible  with  existing  fielded  systems.  This  also  requires  the  ISEA  to 
maintain  some  level  of  certified  pipeline  spares.  As  the  product  evolves,  it  may  be  desirable  to 
upgrade  systems  to  eliminate  incompatibilities  with  future  spares,  or  to  modify  the  architecture 
slightly  to  adopt  a  new  or  modified  standard  reflected  in  the  new  spares.  On  the  other  hand,  changes 
may  affect  a  hardware-software  interface  that  may  require  changing  software  each  time  the  spares 
are  delivered.  The  system  configuration  must  be  maintained  at  current  levels,  documentation  and 
training  must  be  kept  current  and  consistent  with  the  new  elements,  and  any  changed  specifications 
must  be  evaluated.  These  functions  require  an  engineering  oversight  to  the  normal  supply  function. 
This  has  proven  to  be  the  most  effective  means  of  handling  dynamic  spares.  It  is  essential  that  the 
market  survey  ascertain  if  dynamic  spares  are  involved.  Also,  a  notification  clause  should  be  used  in 
procurement  contracts  to  require  supplier  notification  whenever  a  change  is  made  in  the  production 
item. 

Dynamic  spares  are  also  facilitated  by  strong  configuration  management  employing  an  installa¬ 
tion-specific  product  breakdown  based  specification  control  scheme  as  described  as  a  best  practice 
under  the  CONFIGURATION  MANAGEMENT  (CM)  ISSUES. 

HUMAN  FACTORS 

Humans  participate  as  an  integral  part  of  a  system  to  perform  functions  for  which  it  is  literally 
impossible  to  design  a  product  solution.  Humans  are  elegantly  designed  to  deal  with  the  real  world 
and  can  also  be  viewed  as  the  ultimate  in  NDI.  Nobody  can  “design”  or  redesign  a  human  for  a  par¬ 
ticular  system  application,  and  the  humans  available  to  perform  the  system  functions  are  highly  vari¬ 
able  in  skills  and  abilities.  The  human  factors  design  of  the  system  and  its  component  products 
together  with  training  largely  form  the  basis  of  human  performance  as  part  of  a  system. 

Human  factors  can  be  very  challenging  to  properly  integrate  into  NDI  systems.  The  system 
designer  is  faced  with  incorporating  human  factors  in  order  to  reach  the  system  proficiency  goals, 
safety  criteria,  and  cost  goals,  but  NDI  products  already  incorporate  human  factors  elements  that  are 
probably  inconsistent  with  the  standards  already  established  for  the  system  application  environment. 
Since  the  military  has  been  in  the  forefront  of  human  factors  technology  in  many  specialty  areas  for 
many  years,  most  of  the  application  standards  for  human  factors  have  been  very  well  established, 
being  entrenched  in  established  rate  training  and  years  of  use.  On  the  other  hand,  most  commercial 
practices  have  been  established  for  different  application  environments  where  combat  stress  and 
fatigue  are  not  driving  factors.  Furthermore,  commercial  practices  are  just  that — ^practices;  they  are 
not  industry  standards  that  are  well  coordinated  across  the  entire  market.  This  can  result  in  several 
pieces  of  NDI  being  assembled  into  a  system  that  have  different  user  interface  standards.  The  indi¬ 
vidual  pieces  may  have  adequate  user  interfaces,  but  the  inconsistency  across  the  system  cannot  be 
tolerated.  Also,  NDI  may  be  available  from  several  sources,  each  with  their  own  unique  user  inter¬ 
face  (for  instance,  the  different  advanced  controls  between  VCRs  or  microwave  ovens);  this  situation 
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can  lead  to  intolerable  differences  from  system  to  system.  To  cope  with  these  issues,  the  following 
steps  are  recommended: 

1.  Identify  and  document  the  human  factors  standards  that  are  associated  with  the  intended 
applications. 

2.  Identify  and  document  the  other  human  factors  issues  that  are  critical  to  system  proficiency, 
reliability,  and  safety  (including  conditions  of  combat  stress  and  fatigue,  of  potential  operator 
error,  and  of  organizational  maintenance). 

3.  Prioritize  the  issues  and  design  factors  from  steps  1  and  2  above  as  essential,  important,  and 
desired,  and  incorporate  into  the  tailored  screening  criteria. 

4.  Review  potential  candidates  for  conformance  to  essential  standards  and  for  incorporation  of 
imponant  and  desired  features.  Also,  determine  if  design  standards  incorporated  in  the  NDI 
are  in  conflict  with  essential  or  important  criteria.  Use  the  human  factors  criteria  as  part  of  the 
overall  ranking  of  candidates. 

5.  Determine  if  there  are  means  for  bringing  each  candidate  into  conformance  to  essential  stan¬ 
dards. 

6.  Determine  if  there  are  means  for  incorporating  important  or  desired  features  cost  effectively. 

7.  Incorporate  human  factors  criteria  in  the  source  selection  criteria.  Design  and  implement  mod¬ 
ifications  for  selected  candidates  to  maximize  human  factors  compatibility,  as  required. 

The  first  three  steps  are  crucial  to  high-quality  system  designs  and  their  cost-effective  implementa¬ 
tions.  Too  often,  human  factors  criteria  are  disregarded  because  the  NDI  already  has  established 
designs  that  are  difficult  or  impossible  to  change.  However,  the  good  system  designer  can  usually 
work  around  these  limitations  and  still  maintain  the  cost  advantages  of  the  NDI  acquisition.  The 
techniques  available  for  NDI  not  conforming  to  essential  criteria  are  listed  below  from  most  pre¬ 
ferred  to  least  preferred: 

•  Remoting  or  encapsulation. 

•  User  shells. 

•  Repackaging. 

•  Redesign. 

Remoting  uses  the  NDI  in  its  native  form  (unmodified)  but  implements  remote  interface  features  to  a 
user  interface  that  does  conform  to  the  required/desired  standards.  User  shells  provide  a  conforming 
interface  as  an  intermediate  interface  between  the  user  and  the  NDI.  User  shells  are  often  possible 
for  software  applications  where  the  shell  translates  between  the  desired  user  interface  and  the  user 
interface  designed  into  the  application.  Many  commercial  applications  include  macro-language  fea¬ 
tures  to  promote  application  customization  through  user  shells.  Often  the  core/required  functionality 
of  the  NDI  can  be  easily  repackaged  to  conform  to  established  standards.  For  instance,  the  knobs  of 
a  commercial  radar  can  be  changed  to  conform  to  the  functional  shape  standards  of  MIL-STD-1472. 
In  the  extreme,  the  functional  elements  of  the  NDI  might  be  physically  repackaged  in  a  new  housing 
having  conforming  user  interface  standards.  Many  software  applications  written  for  UNIX/POSIX° 
operating  systems  have  the  capability  of  replacing  the  user  interface  with  a  customer-designed  inter¬ 
face,  this  is  equivalent  to  repackaging.  Redesign  is  the  most  radical  step  where  the  user  interface  and 
underlying  design  features  are  modified  to  bring  the  NDI  into  conformance. 
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Human  factors  requirements  are  extremely  important.  Good  human  factors  is  essential  for  good 
system  operation  under  all  conditions.  Good  human  factors  can  also  dramatically  lower  training  and 
maintenance  costs  by  improving  the  organizational  capability  to  interact  properly  with  the  system. 
The  human  factors  requirements  can  have  so  great  a  system  quality  impact  that  they  are  the  primary 
reason  for  modifying  NDI. 


TRAINING 

Training  should  be  considered  integral  to  and  an  essential  part  of  any  system.  NDI  acquisitions  are 
prone  to  miss  important  training  issues  because  there  is  a  strong  tendency  to  focus  on  the  product 
acquisition  alone.  Also,  NDI  acquisitions  are  often  executed  so  rapidly  that  any  training  that  needs 
to  be  developed  often  cannot  go  through  normal  or  recommended  training  development  cycles. 

When  training  must  be  developed  for  a  product  that  can  be  fielded  in  under  6  months,  there  are 
severe  strains  put  on  the  “smart  buyer”  team  executing  the  system  acquisition.  When  training 
already  exists  for  NDI,  numerous  tasks  must  still  be  done  to  integrate  the  training  into  the  entire  sys¬ 
tem  acquisition. 

TRAINING  SITUATION  ANALYSIS 

The  Training  Situation  Analysis  is  fundamental  to  the  determination  of  training  needs  for  a  new 
system.  In  developmental  acquisitions,  the  analysis  is  conducted  after  the  system  design  but  prior  to 
the  Engineering  and  Manufacturing  Development  Phase  so  the  training  can  be  developed  while  the 
products  are  in  detailed  design.  In  NDI  acquisitions,  the  analysis  must  be  conducted  concurrently 
and  integrated  with  the  system  design  since  the  analysis  partially  depends  on  system  partitioning 
decisions.  However,  there  will  still  be  training  analysis  effort  required  after  the  system  design  is 
complete.  The  purpose  of  the  training  situation  analysis  is  to  determine  what  skills  are  needed  for 
the  system  and  for  system  component  operation  and  maintenance,  what  existing  training  supports 
these  requirements,  what  existing  training  (ranging  from  formal  classroom  training  to  on-the-job 
training)  may  be  impacted  by  the  new  system,  what  new  training  is  required,  what  NDI  training  may 
exist  to  support  the  new  system,  what  training  costs  exist,  and  what  training  resources  need  to  be 
acquired  or  developed  to  support  the  system. 

For  integrated  NDI  and  major  modifications,  enough  time  is  usually  available  for  the  training  anal¬ 
ysis  and  any  training  development  to  be  done  while  the  modifications  and  integration  are  being 
designed  and  executed.  This  allows  the  training  to  be  validated  and  introduced  through  relatively 
normal  procedures.  However,  any  training  requiring  development  must  usually  be  designed  and 
delivered  on  an  accelerated  schedule. 

Other  NDI  being  acquired  as  system  or  major  subsystem  will  normally  have  NDI  training  as  well. 
During  the  market  analysis,  a  training  screen  should  be  incorporated  in  the  assessment  of  potential 
candidates.  The  tailored  screening  process  can  include  the  validation  of  NDI  training  as  a  part  of  the 
product  demonstration.  Alternatively,  the  NDI  training  can  be  validated  during  the  source  selection 
process  or  as  a  part  of  the  first  article  acceptance;  however,  earlier  is  better  as  it  allows  for  modifica¬ 
tions  that  might  be  required.  If  the  existing  training  is  adequate,  it  should  be  acquired  integrally  to 
the  NDI  system  acquisition,  incorporating  any  modifications  determined  to  be  necessary. 

Other  NDI  being  acquired  as  products  assembled  into  a  system  should  also  have  a  training  screen 
incorporated  into  the  assessment  of  potential  candidates.  However,  system-level  training  will  need  to 
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be  developed  in  addition  to  any  product-level  training  that  might  be  acquired  with  the  NDI.  If 
resources  are  limited,  sufficient  training  analysis  needs  to  be  done  to  allow  the  design  and  conduct  of 
this  system-level  training  concurrent  with  the  fielding  of  the  system,  even  if  other  analysis  tasks  need 
to  be  deferred.  The  system-level  training  needs  to  cover  those  issues  above  the  product  (black  box) 
level,  plus  any  operation  and  maintenance  elements  at  the  product  level  that  have  system-level 
implications.  These  types  of  acquisitions  seldom  have  enough  schedule  available  for  validation  of 
training  or  documentation,  so  the  post-fielding  assessment  activities  become  very  important  to  the 
meeting  of  overall  system  quality  goals.  Supplemental  training  after  fielding  must  be  planned  and 
funded  as  well,  since  the  system-level  training  will  not  touch  all  of  the  issues  needed  for  adequate 
system  use  and  field  maintenance.  It  may  also  be  necessary  to  retrain  the  initial  field  users  and  main- 
tainers  after  the  final  training  packages  are  ready. 

All  NDI  acquisitions  should  include  a  post-fielding  training  assessment  to  allow  emergent  issues  to 
be  discovered  and  incorporated  into  the  full  system  training  package. 

SYSTEM-LEVEL  TRAINING 

There  is  a  large  tendency  to  focus  on  product-level  training  in  NDI  acquisitions  to  the  detriment  of 
system-level  training.  This  is  especially  true  of  NDI  modifications  to  existing  systems  and  for  NDI 
system  upgrades.  The  result  is  for  training  to  be  included  for  each  individual  piece  of  the  system  but 
for  many  system-level  issues  to  be  omitted.  The  most  difficult  of  issues  to  discover  are  those  that 
anse  from  new  modes  and  capabilities  introduced  by  a  “black  box”  (or  new  software  application). 
The  training  might  be  available  for  the  operation  and  maintenance  of  the  “box,”  but  the  implications 
of  the  new  capability  at  the  system  level  will  not  be  understood. 

Software  trouble  reports  have  been  made  against  fielded  NDI  systems  where  no  software  existed 
simply  because  system  training  and  system  documentation  were  inadequate.  The  field  personnel 
would  not  be  able  to  bring  up  the  system  because  equipment  were  in  different  modes  (unknown  to 
the  operators/maintainers),  yet  check  out  each  box  only  to  find  it  was  properly  functioning.  The  pre¬ 
sumption  was  *at  there  must  be  some  piece  of  software  causing  the  problem.  In  fact,  it  was  a  failure 
of  the  system  life-cycle  maintenance  function  to  identify,  document,  and  provide  training  to  a  new 
system  mode  that  was  not  covered  by  system-level  training  or  system  documentation. 

TRAINING  ASSESSMENT  SCREEN 

A  training  assessment  screen  should  be  done  as  an  integral  part  of  the  market  survey.  The  training 
screen  should  not  be  used  to  eliminate  potential  candidates;  however,  it  should  be  used  to: 

•  identify  existing  training  associated  with  each  candidate, 

•  evaluate  the  methods  of  training  delivery  employed, 

•  document  training  issues  at  the  NDI  product  level, 

•  develop  information  on  NDI  training  costs,  and 

•  determine  what  training  resources  may  be  required,  peculiar  to  the  NDI. 

The  training  screen  thus  includes  many  of  the  elements  of  the  training  situation  analysis  tailored  to 
each  NDI  candidate.  The  information  developed  by  a  training  screen  is  critical  to  the  development 
of  suitable  training  within  the  compressed  schedules  commonly  encountered  in  NDI  acquisitions. 
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INITIAL  TRAINING 


Initial  training  for  NDI  systems  should  be  planned  prior  to  and  integrated  with  each  system  instal¬ 
lation.  It  is  common  for  each  NDI  installation  to  be  slightly  different,  so  it  is  important  to  document 
and  train  to  these  differences.  If  the  system  is  to  be  supported  by  formal  classroom  training,  it  is 
very  common  for  this  training  to  not  be  available  until  the  NDI  system  has  been  in  the  field  for  sev¬ 
eral  years,  so  it  is  critical  that  initial  training  be  planned  to  cover,  even  overlap,  this  formal  classroom 
training.  In  addition,  the  system  upgrades  and  overhauls  that  will  occur  during  the  system  life  will 
require  initial  training  even  if  formal  classroom  training  is  on  line. 

Generally,  formal  classroom  training  will  not  be  the  primary  mode  of  training  delivery  for  most 
NDI  systems.  Classroom  training  may  be  the  best  mode  for  delivering  the  overview  information  and 
system-level  training,  but  much  of  the  product-level  operation  and  maintenance  training  may  be  so 
“box”  specific  that  only  embedded  or  on-site/on-the-job  training  will  be  sufficient.  A  system  planner 
should  expect  that  the  effort  to  develop  quality  training  for  an  NDI  system  will  be  an  appreciable 
portion  of  the  overall  system  support  effort. 

TRAINING  DOCUMENTATION 

All  training  information,  whether  procured  or  developed,  should  be  translated  into  an  electronic 
medium  (preferably  a  CALS-compatible  format).  NDI  systems  tend  to  be  highly  volatile  because  the 
product  lives  are  relatively  short  compared  to  the  system  life,  making  system  upgrades  necessary. 
Also,  there  may  be  a  multitude  of  different  system  configurations.  As  a  result,  the  training  packages 
might  need  to  be  tailored  to  each  configuration  and  become  site  dependent.  In  any  case,  training 
documentation  needs  to  be  integrated  with  the  installed  configurations  and  maintained  under 
common  configuration  control.  The  electronic  maintenance  of  training  allows  the  training  to  be 
updated  effectively  and  efficiently  on  a  site-by-site  basis,  as  required. 

Post-Fielding  Training  Assessment 

A  post-fielding  training  assessment  is  a  good  idea  for  any  complex  system.  However,  NDI  sys¬ 
tems  can  be  complex  while  appearing  to  be  simple;  therefore,  a  post-fielding  assessment  should  be 
done  in  any  case  to  ensure  system  quality  and  to  control  risks.  Ideally,  a  post-fielding  assessment 
should  take  at  least  the  following  “snapshots”  of  the  system: 

•  Within  30  days  of  installation  or  during  exercises  or  refresher  training. 

•  After  4  months  of  use  or  immediately  prior  to  deployment. 

•  After  1  year  of  use  or  immediately  after  deployment. 

It  is  especially  critical  to  capture  information  immediately  after  a  system  deployment  for  Navy  ship 
installations  because  there  is  often  a  crew  rotation  shortly  after  the  return  from  deployment,  and  the 
information  will  be  lost  if  the  assessment  does  not  capture  it  prior  to  their  leaving.  (An  assessment 
team  might  even  be  sent  to  ride  a  ship  back  from  deployment  in  order  to  ensure  that  the  information 
is  captured.)  The  goals  of  the  assessment  include  the  following: 

•  Assess  the  adequacy  of  existing  training. 

•  Determine  if  there  are  any  additional  issues  that  should  be  covered  by  training. 

•  Determine  the  accuracy  of  training  provided,  especially  compared  to  actual  system  use. 
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•  Assess  the  integration  of  training  with  the  overall  system  and  with  other  system  support  ele¬ 
ments. 

Following  these  assessments,  the  project  should  determine  if  additions,  corrections,  or  other  modifi¬ 
cations  are  needed  to  existing  training.  The  project  should  also  be  prepared  to  conduct  supplemental 
training  if  added  issues  are  discovered  or  if  major  modifications  to  the  training  package  are  indicated. 


DOCUMENTATION 

Most  of  the  issues  relating  to  documentation  are  discussed  in  the  DATA  ACQUISITION  section; 
however,  two  special  issues  are  discussed  here— readability  and  completeness.  Several  forms  of  doc¬ 
umentation  normally  create  subtle  problems  in  NDI  acquisitions,  especially  COTS  acquisitions: 
operator  and  maintenance  technical  manuals,  software  interface  control  documents  (such  as  Interface 
Design  Documents  or  the  interface  section  of  a  System  Design  Document),  and  installation  and  spec¬ 
ification  control  drawings. 

Technical  manuals  associated  with  COTS  are  written  for  the  market  requirements.  COTS  mainte¬ 
nance  manuals  are  commonly  written  for  a  reading  grade  level  far  above  the  service  standard  for  the 
designated  operator  and  maintenance  personnel.  (Most  large  city  newspapers  use  a  fifth-grade  read¬ 
ing  level,  which  is  below  virtually  all  service  standards.)  COTS  manuals  often  contain  company- 
unique,  product-unique,  or  market-unique  jargon.  (For  instance,  a  speech  processor  used  in  the 
music  industry  may  have  terms  unique  to  musicians  familiar  with  the  technology.  When  used  for  a 
military  communication  system,  the  operators  and  maintainers  are  unlikely  to  know  the  special  ter¬ 
minology  used  in  the  technical  documentation.)  This  implies  rewriting  the  technical  manual.  On  the 
other  hand,  some  MILOTS  operator  manuals  written  for  other  service  requirements  are  written  very 
far  below  the  standards  for  the  target  personnel,  resulting  in  a  manual  that  does  not  communicate 
well.  These  manuals  may  also  require  modification.  Manuals  for  foreign  items  may  require  transla¬ 
tion,  and  the  translation  may  need  to  be  rewritten  to  make  it  more  readable.  In  all  cases,  it  is  impor¬ 
tant  for  the  manuals  to  be  written  concisely  and  clearly  for  the  target  population,  taking  into  account 
that  the  target  population  may  have  less  training  and  fewer  skills  than  what  was  desired  by  the  sys¬ 
tem  designers.  Obtaining  the  manuals  in  an  electronic  form  (even  unformatted  text  plus  pictures), 
promotes  the  cost-effective  translation  of  the  information  into  a  suitable  form  for  delivery  to  the 
users  and  maintainers.  This  allows  some  of  the  powerful  computer-based  tools  to  be  applied  to  aid  in 
tailoring  the  readability  as  well  as  creating  interactive  electronic  technical  manuals  and  embedded 
online  documentation. 

Software  interface  documentation  is  notoriously  difficult  to  “get  right.”  Very  often,  NDS  interface 
documentation  is  limited  to  a  small  paragraph  on  the  side  of  the  shipping  box.  Even  GOTS  software 
is  usually  deficient  in  specifying  the  interface  requirements.  The  primary  difficulty  is  identification 
of  the  possible  interfaces  in  an  open  architecture  environment.  A  specific  software  application  may 
need  certain  resources  to  load  and  mn  effectively,  and  the  accumulation  of  applications  running  con¬ 
currently  may  still  be  well  within  the  advertised  capacity  of  the  computer  resources.  But  one 
application  may  still  demand  specific  resources  at  the  same  time  as  another  and  cause  the  system  to 
crash,  hang,  create  errors,  or  respond  in  other  unreliable  ways.  It  is  almost  always  a  requirement  to 
include  stress  tests  of  hardware-software  combinations  as  a  part  of  the  screening  process  in  order  to 
determine  what  unreliable  behaviors  there  may  be  and  what  the  conditions  are  that  cause  these 
behaviors.  When  such  behaviors  are  discovered,  the  existing  interface  documentation  must  be 
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modified  or  supplemented  to  avoid  the  problems.  Additional  system  management  software  may  be 
needed  to  mediate  the  potential  problems  that  cannot  be  operationally  tolerated. 

Once  the  issue  of  completeness  is  resolved,  software  interface  documentation  must  also  be 
checked  for  readability.  In  this  case,  readability  and  clarity  of  the  requirements  documented  are  both 
essential.  It  is  a  good  practice  to  perform  a  formal  inspection  on  all  software  interface  documenta¬ 
tion  prior  to  its  acceptance.  In  fact,  a  formal  inspection  may  be  performed  prior  to  resolving  the 
completeness  and  accuracy  issues  noted  above,  and  a  second  inspection  performed  afterward.  Since 
the  documentation  for  software  interfaces  is  increasingly  being  produced  in  computer-aided  software 
engineering  environments,  it  is  available  in  electronic  form.  However,  there  are  no  current  industry- 
recognized  standards  for  the  database  form  of  this  kind  of  documentation,  so  the  preferred  electronic 
format  supplied  may  not  be  compatible  with  the  tools  available  to  the  system  software  support  activ¬ 
ity  tools.  To  ensure  tool  readability  as  well  as  human  readability,  it  is  usually  necessary  to  conduct 
some  data  transfer  experiments  until  the  right  combination  of  file  formats  is  found  to  make  the  data 
readable  without  losing  any  information.  Drawing  documentation  has  a  similar  problem,  as  noted 
below. 

Control  drawings  are  also  problematical  for  most  NDI.  Interface  and  specification  control  drawing 
tend  to  be  incomplete  and  may  make  references  to  company  standard  processes  or  procedures  or 
specifications  that  are  not  part  of  the  disclosure  package.  It  is  important  to  review  critical  documents 
for  completeness  and  to  ensure  that  all  information  needed  to  interpret  the  drawings  is  made  avail¬ 
able.  In  addition,  the  documentation  can  be  validated  through  use  by  a  third  party  (such  as  an  instal¬ 
lation  activity  or  In-service  Engineering  Agent).  Formal  inspections  are  useful  to  ensure  complete¬ 
ness,  and  should  be  conducted  prior  to  acceptance  of  the  documentation. 

When  drawings  are  pres  ided  in  an  electronic  medium,  it  is  important  to  run  tests  of  the  readability. 
Many  of  the  applications  used  to  create  the  drawings  use  proprietary  file  formats  that  are  translated 
in  order  to  be  delivered  in  a  preferred  electronic  format.  Data  can  be  lost  or  changed  through  the 
translation  process,  even  by  translators  written  by  the  supplier  of  the  application.  By  performing  test 
exchanges,  any  problems  can  be  discovered  early  and  workarounds  or  fixes  can  be  identified.  Also, 
while  it  is  desirable  to  obtain  all  data  in  CALS-compatible  file  formats,  it  is  often  necessary  to 
receive  the  data  in  a  commercial  standard  and  to  convert  it  to  a  CALS  format. 

SYSTEM  DOCUMENTATION 

System-level  documentation  often  does  not  exist  for  new  or  modified  systems  employing  NDI. 
Instead,  there  will  be  an  assembly  of  the  documentation  for  all  of  the  products  that  make  up  the  sys¬ 
tem.  System  documentation  may  exist  only  for  installation  and  checkout  purposes,  but  be  missing  for 
mission  application  and  maintenance.  These  circumstances  arise  because  the  system  is  designed  out 
of  NDI  pieces,  and  documentation  already  exists  for  each  piece  of  NDI.  However,  the  existing  docu¬ 
mentation  may  not  be  appropriate  for  the  system  purposes  or  may  omit  critical  system  usage 
information  such  as  concepts  of  operation,  concepts  of  support,  and  system-level  maintenance. 
Numerous  systems  have  been  fielded  with  this  deficiency,  causing  excessive  trouble  reports  and 
expensive  but  avoidable  maintenance  actions.  For  NDI,  it  is  especially  critical  that  each  system 
upgrade  also  be  covered  by  revised  system  documentation,  since  the  upgrade  probably  introduces 
new  modes  and  capabilities  that  are  previously  undocumented.  Even  if  a  capability  of  a  piece  of  NDI 
is  not  being  used,  its  existence  and  the  fact  that  it  is  not  used  should  be  documented  in  the  system 
documentation.  This  circumstance  can  result  in  a  need  for  the  system  documentation  to  be  site 
dependent.  If  this  is  the  case,  the  system  documentation  must  be  carefully  controlled  as  a 
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site-dependent  configuration  item.  The  cost-effective  and  efficient  maintenance  of  system  documen¬ 
tation  is  greatly  enhanced  by  maintaining  it  in  an  electronic  format.  The  system  documentation  must 
also  be  thoroughly  reviewed  for  accuracy  each  time  a  change  is  made  to  any  configuration  item  in 
the  system,  as  there  may  be  new  issues  introduced  to  the  system  level  from  lower  level  product 
changes. 

System  documentation  and  supporting  product  documentation  should  be  reviewed  for  accuracy 
and  utility  as  part  of  a  post-fielding  assessment  for  all  NDI  systems.  (See  Post-Fielding  Training 
Assessment.)  Attention  to  the  generation  and  maintenance  of  quality  system-level  documentation 
promotes  higher  system  quality  in  the  field  and  makes  system-level  training  much  easier  to  teach  and 
maintain. 


SOFTWARE  ISSUES 

Several  different  classes  of  software  are  encountered  in  NDI  systems.  Clearly,  there  is  Non-Devel- 
opmental  Software  (NDS)  and  its  major  subset,  Commercial  Off-The-Shelf  Software  (COTSS). 
Modification  software  is  that  software  needed  to  adapt  COTSS  or  other  NDS  into  a  new  application. 
Modification  software  is  distinct  from  NDS  that  requires  modification  for  application  to  a  system^ 
NDS  requiring  modification  is  simply  modified  software  (although  the  modification  is  restricted  to 
be  less  than  a  30-percent  change  as  measured  in  source  lines  of  code  or  some  similar  metric). 
Integration  software  is  the  software  “glue”  used  to  functionally  link  items  within  a  system;  most 
integration  software  is  newly  developed  for  the  specific  application.  Other  newly  developed  soft¬ 
ware,  modification  software,  and  integration  software  can  be  managed  as  a  single  set. 

In  addition  to  the  traditional  types  of  software — applications,  databases,  and  operating  systems — it 
is  necessary  to  define  several  subsets  of  application  software  as  distinct  types.  Hidden  software  is 
code  buried  within  an  off-the-shelf  configuration  item  that  is  not  visible  at  the  configuration  item 
interface.  Interface  software  directly  implements  interfaces  to  existing  external  standards.  Interac¬ 
tive  software  only  partially  implements  the  functionality  of  a  particular  configuration  item  and  is  vis¬ 
ible  at  the  interfaces  to  the  configuration  item.  Application  software  then  assumes  a  more  limited 
definition — that  software  fully  implementing  a  system  function  and  constituting  a  self-contained 
configuration  item  (i.e.,  a  computer  software  configuration  item  [CSCI]).  Each  of  the  software  types 
can  be  represented  by  any  of  the  above  classes.  This  creates  a  matrix  of  requirements  for  the  acquisi¬ 
tion  and  life-cycle  management  of  software  as  given  in  table  4. 
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Hidden 

Software 


Interactive 

Software 


Application 

Software 


Operating 

Systems 

(OS) 


Interface 

Software 


Table  4.  Software  acquisition  and  maintenance  strategies. 


NDS,  incl.  COTSS 

Modified  Software 

Integration  Software 

Acquisition — Obtain  existina 
documentation  as  provided 
with  existing  product  package 
Maintenance — All  source 
support  (license  agreements, 
warranties,  contractor  service 
agreements).  ISEA  or  LCMA 
monitors  configuration  status. 

Acquisition — Obtain  existing 
documentation  as  provided 
with  existing  product  package 
Maintenance — All  source 
support  (license  agreements, 
warranties,  contractor  service 
agreements).  ISEA  or  LCMA 
monitors  configuration  status. 

Acquisition — Obtain  existing 
documentation  as  provided 
with  existing  product  package 
Maintenance — ^All  source 
support  (license  agreements, 
warranties,  contractor  service 
agreements).  ISEA  or  LCMA 
monitors  configuration  status. 

Acquisition — Obtain  existing 

Acquisition — Obtain  existing 

Acquisition — Obtain  existing 

documentation  and  product 
Maintenance — All  source 
support  (license  agreements, 
warranties,  contractor  service 
agreements).  ISEA  or  LCMA 
monitors  configuration  status. 

documentation  and  product 
Maintenance — ^All  source 
support  (license  agreements, 
warranties,  contractor  service 
agreements).  ISEA  or  LCMA 
monitors  configuration  status. 

documentation  and  product. 
Acquiring  activity  performs 
limited  quality  reviews 

Maintenance — ^All  source 
support  (license  agreements, 
warranties,  contractor  service 
agreements).  LCMA  CM. 

Acquisition — Obtain  existing 

Acquisition — Obtain 

Acquisition — Obtain 

documentation  and  product 
Maintenance — All  source 
support  (license  agreements, 
warranties,  contractor  service 
agreements) 

Maintenance  Plan, 
requirements  and  design 
documents,  VDD,  and  code. 
Acquirer  performs  limited 
quality  reviews 

Maintenance — SSA  organic 
support  and  LCMA  CM 

Maintenance  Plan, 
requirements  and  design 
documents,  VDD,  and  code. 
Acquirer  performs  full  quality 
reviews  and  audits. 

Maintenance — SSA  organic 
support  and  LCMA  CM 

Acquisition — Obtain  existing 
documentation  and  product 
Maintenance — All  source 
support  (license  agreements, 
warranties,  contractor  service 
agreements) 

Acquisition — Obtain 
Maintenance  Plan, 
requirements  and  design 
documents,  VDD,  and  code. 
Acquirer  performs  limited 
quality  reviews 

Maintenance — SSA  organic 
support  and  LCMA  CM 

Acquisition — Obtain 
Maintenance  Plan, 
requirements  and  design 
documents,  VDD,  and  code. 
Acquirer  performs  full  quality 
reviews  and  audits. 
Maintenance — SSA  organic 
support  and  LCMA  CM 

Acquisition — Obtain  existina 

Acquisition — Obtain 

Acquisition — Obtain 

documentation  and  product. 
Maintenance — All  source 
support  (license  agreements, 
warranties,  contractor  service 
agreements) 

Maintenance  Plan, 
requirements  and  design 
documents,  VDD,  and  code. 
Acquirer  performs  limited 
quality  reviews 

Maintenance — SSA  organic 
support  and  LCMA  CM 

Maintenance  Plan, 
requirements  and  design 
documents,  VDD,  and  code. 
Acquirer  performs  full  quality 
reviews  and  audits. 

Maintenance — SSA  organic 
support  and  LCMA  CM 

Acquisition — Obtain  existing 

Acquisition — Obtain 

Acquisition — Obtain 

documentation  and  product. 
Acquiring  activity  performs 
limited  quality  reviews 

Maintenance — Source  or 
contractor  support  with 

LCMA  CM 

Maintenance  Plan, 
requirements  and  design 
documents,  VDD,  and  code. 
Acquirer  performs  limited 
quality  reviews 

Maintenance — SSA  oraanic 
support  and  LCMA  CM 

Maintenance  Plan, 
requirements  and  design 
documents,  VDD,  and  code. 
Acquirer  performs  full  quality 
reviews  and  audits. 

Maintenance — SSA  organic 
support  and  LCMA  CM 
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The  primary  issues  for  acquisition  are  (1)  the  types  of  documentation  to  be  acquired  with  the  prod¬ 
uct  and  (2)  the  extent,  if  any,  of  quality  reviews  or  IV&V  to  be  done.  For  maintenance,  the  issues  are 
(1)  who  is  responsible  for  maintenance  and  (2)  who  performs  configuration  management.  Since  hid¬ 
den  and  interactive  software  is  intimately  tied  to  the  hardware  product,  these  forms  of  software  are 
best  handled  together  with  the  hardware  as  a  single  configuration  item.  Interactive  integration  soft¬ 
ware  usually  requires  some  degree  of  quality  oversight  because  of  its  complex  interactions  with  other 
system  components.  NDS,  including  COTSS,  products  should  be  handled  as  “shrink-wrapped” 
products  complete  with  support;  special  service  agreements,  perhaps  with  a  licensed  third  party,  can 
be  negotiated  when  the  normally  available  support  is  not  adequate.  Many  systems  will  contain  a  mix 
of  these  software  classes  and  types,  so  procurement  documents  must  provide  for  the  flexibility  in 
dealing  with  the  full  range  of  variability  in  terms  of  configuration  management  and  quality  manage¬ 
ment  activities  and  tasks  plus  the  acquisition  of  the  needed  level  of  information.  Even  though  modi¬ 
fication  software  and  integration  software  require  essentially  the  same  documentation,  the  extensive¬ 
ness  of  the  documentation  for  modification  software  may  be  limited  to  only  those  elements 
associated  with  the  actual  modification  plus  the  interfacing  elements. 


ROLES  IN  NDI  LIFE-CYCLE  ACQUISITION  AND  MANAGEMENT 

Buying  NDI  raises  issues  that  dictate  some  new  definitions  to  the  traditional  roles  in  the  life-cycle 
management  of  systems.  There  are  significant  differences  in  the  requirements  for  planning  for  the 
system  life  cycle  of  COTS-based  systems  from  developmental  systems  that  have  implications  in  the 
types  of  expertise  required  for  the  operations  and  support  phase  management,  in  the  funding  needed 
to  support  technology  upgrades,  and  in  the  facilities  needed  to  maintain  configuration  control.  Fur¬ 
thermore,  the  rapid  changes  in  system  configurations  characteristic  of  NDI  systems  dictate  closer 
relationships  and  coordination  between  the  major  players  in  the  life-cycle  support  of  the  systems. 
Continuing  organizational  relationships  require  long-term  agreements  and  funding  commitments  that 
can  be  difficult  to  establish  contractually  or  to  fund  in  eras  of  shrinking  budgets.  This  section  dis¬ 
cusses  these  issues  and  makes  recommendations  based  on  the  NDI  program  relationships  that  have 
been  most  successful. 

The  roles  of  the  various  managers  and  agents  are  defined  in  various  Systems  Command  instruc¬ 
tions.  These  instructions  vary  in  the  titles  assigned  to  the  various  roles  but  are  in  practical  harmony 
with  regard  to  the  issues  discussed  herein. 

ACQUISITION/SYSTEM  MANAGER 

The  Acquisition  or  System  Manager  (“Manager”)  is  the  designated  representative  of  the  Systems 
Command  or  Program  Office  responsible  for  planning  and  executing  the  acquisition  from  the  docu¬ 
mentation  of  the  requirements  through  its  introduction  into  operational  use.  This  includes  the  overall 
program  management  and  technical  direction  of  engineering,  including  capability,  assurance  require¬ 
ments  and  assessment,  maintenance  engineering,  configuration  management,  and  logistics  support. 
The  Manager  has  the  responsibility  of  planning  and  implementing  the  system  life-cycle  support, 
including  pre-planned  product  improvements  (P^I),  consistent  with  applicable  laws,  regulations,'  and 
policies.  While  the  overall  responsibility  lies  with  the  Manager,  engineering  agents  are  available  to 
do  the  detailed  work  to  achieve  the  program  goals  established  by  the  Manager."  The  following  NDI 
issues  are  important  to  the  Acquisition/System  Manager: 
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•  Funding — for  the  entire  acquisition  phase  and  into  the  operations  and  support  phase.  Program 
decisions  should  be  life-cycle  cost-based.  For  NDI  systems,  the  funding  of  life-cycle  manage¬ 
ment  activities  and  technological  upgrades  are  critical.  An  entire  product  change-out  may 
occur  in  5  years  or  less. 

•  Acquisition  support  activities — the  acquisition  team  must  provide  “smart  buyer”  capabilities 
and  conduct  the  market  surveys  needed  to  make  accurate  decisions.  This  normally  involves 
forming  an  integrated  team  with  representatives  of  R&D,  ISEA,  SSA,  training,  and  life-cycle 
management  activities.  The  support  activities  must  also  be  in  a  position  to  address  the  issues 
that  have  been  identified  in  this  document. 

•  Acquisition  Planning — acquisition  plans  must  be  constructed  to  maintain  an  optimum  competi¬ 
tive  environment.  NDI  systems  risk  forcing  sole  source  support  of  high-level  equipments  or 
subsystems  (in  contrast  to  components  or  minor  assemblies)  of  considerable  value  unless  a 
life-cycle  management  support  infrastructure  is  established. 

•  Procurement  Planning — source  selection  plans  should  take  advantage  of  the  information  avail¬ 
able  and  the  capabilities  that  can  be  demonstrated  with  NDI  systems  while  also  accounting  for 
the  potential  gaps  and  low  confidence  levels  associated  with  some  of  the  information  needed 
for  support  planning  of  NDI. 

•  Support  planning — support  plans  need  to  provide  for  the  flexibility  to  use  contractor  support 
services  and/or  warranty  services  while  maintaining  high  levels  of  operational  availability  and 
to  account  for  the  issues  of  many  rapidly  changing  configurations. 

ASSURANCE  ENGINEERING  MANAGER  (AEM) 

The  Assurance  Engineering  Manager  (AEM)  or  Logistics  Manager  is  the  designated  representative 
of  the  Systems  Command  or  Program  Office  assigned  to  support  the  Manager  in  system  assurance 
issues  and  is  responsible  to  the  Manager  for  the  interpretation  and  effective  implementation  of  DoD 
and  Systems  Command  policies  for  reliability,  maintainability,  availability,  supportability,  sustain¬ 
ability,  safety,  and  quality.  AEM  responsibilities  extend  throughout  the  life  of  the  system.  Primarily, 
the  AEM  is  responsible  for  coordinating  system  assurance  issues  among  the  engineering  agents, 
ensuring  that  system  assurance  issues  are  properly  represented  in  system  documentation  and  procure¬ 
ment  requirements,  providing  for  the  conduct  of  appropriate  testing  to  verify  that  system  assurance 
requirements  have  been  met,  developing  affordable  and  effective  support  plans,  and  ensuring  that 
systems  assurance  issues  are  addressed  appropriately  at  milestone  decision  reviews  and  other  signifi¬ 
cant  acquisition  reviews.  The  following  NDI  issues  are  important  to  the  Assurance  Engineering  Man¬ 
ager: 

•  Determining  and  implementing  support  requirements  in  order  to  achieve  the  required  opera¬ 
tional  availability.  Many  support  constraints  imposed  by  the  commercial  support  of  COTS  are 
not  consistent  with  high  levels  of  operational  availability  unless  special  support  features  are 
built  into  the  system. 

•  Implementing  effective  support  plans  that  address  the  issues  raised  in  this  document,  especially 
the  rapidly  changing  internal  configurations  of  many  NDI  products  that  are  uncontrollable  by 
the  program. 

•  Making  affordable  and  effective  support  decisions  based  on  lower  confidence-level  informa¬ 
tion. 
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TECHNICAL  DIRECTION  AGENT  (TDA) 

As  defined  by  Systems  Command  instructions,  a  Technical  Direction  Agent  (TDA)  is  a  DoD  agent 
with  the  charter  to  serve  as  the  director  of  systems  engineering  activities  and  responsible  for  imple¬ 
menting  the  tasks  defined  by  DoD  Instruction  5000.2.  TDA  responsibilities  may  be  retained  by  the 
cognizant  Systems  Command  or  assigned  to  a  field  activity,  such  as  a  Navy  Laboratory AVarfare  Cen¬ 
ter.  In  either  case,  the  TDA  assists  the  Manager  in  establishing  system  concepts,  defining  a  technical 
approach,  defining  system  requirements  (including  procurement  requirements),  performing/directing 
research,  development,  tests,  and  simulations  to  investigate  technical  issues,  probing  alternative  tech¬ 
nical  approaches,  and  evaluating  design  agent  achievements.  The  TDA  must  oversee  the  conduct  of 
the  market  research  functions  and  define/approve  the  acceptable  standards  criteria  to  be  used  in  the 
system  partitioning.  The  TDA  must  also  coordinate  the  SDA,  DA,  SIA,  AEA/ISEA,  SSA,  STA,  and 
LCMA  activities  and  taskings  through  the  acquisition  phase.  The  following  NDI  issues  are  imoortant 
to  the  TDA: 

•  Analyzing  market  research/implications  of  the  findings. 

Defining  procurement  requirements  that  are  performance  based  and  that  cite  appropriate  inter¬ 
face  standards. 

•  Defining  screening  requirements. 

•  Assessing  information  needs  and  confidence  levels. 

SYSTEM  DESIGN  AGENT  (SDA) 

The  System  Design  Agent  (SDA)  is  responsible  for  transforming  operational  requirements  into  a 
preferred  technical  approach  and  for  performing  system  partitioning  functions  down  to  the  level  that 
performance  specifications  can  be  prepared  for  tasking  a  design  agent  or  making  a  procurement.  The 
SDA  may  be  a  DoD  agent  or  a  contractor.  When  the  SDA  is  a  DoD  agent,  the  SDA  is  often  an 
extension  of  the  TDA  function.  If  the  TDA  is  retained  by  the  Systems  Command,  the  SDA  is  often  a 
field  activity  tasked  by  the  TDA.  It  is  also  possible  for  the  top-level  SDA  functions  (interpretation  of 
operational  requirements  and  top-level  system  partitions)  to  be  performed  “in-house”  by  a  DoD 
agent  and  the  remaining,  lower  levels  of  the  system  partitioning,  to  be  performed  by  a  contractor.  If 
the  SDA  is  a  contractor,  the  contract  should  be  administered/technically  controlled  by  the  TDA.  The 
following  NDI  issues  are  important  to  the  SDA: 

•  Managing  system  requirements  in  the  acquisition  phase,  including  P^I.  This  is  especially  criti¬ 

cal  for  requirements  stemming  from  the  system  being  a  part  of  a  larger  set  of  systems  formin'^ 
an  operational  capability.  “ 

•  System  partitioning  consistent  with  potential  NDI. 

•  System  partitioning  consistent  with  market  research  results. 

•  System  partitioning  consistent  with  preferred/acceptable  interface  standards. 

Documenting  interface  standards,  including  the  variable  flavors  encountered  in  commercial 
standards.  Also,  the  potential  problems  in  documenting  market-driven  commercial  standards 
or  company  proprietary  standards. 

Selecting  interface  standards  consistent  with  the  likely  evolution  technologies. 

•  Including  system  assurance  issues  in  the  overall  system  architecture. 
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The  interactions  between  these  issues  often  cause  communications  problems  between  the  SDA  func¬ 
tion  and  the  other  systems  engineering  functions  being  directed/coordinated  by  the  AEM  and  TDA. 
Also,  the  choice  between  a  DoD  agent  or  contractor  functioning  as  SDA  should  not  be  made  on  the 
basis  of  “we’ve  always  done  it  this  way.”  Systems  that  have  critical  or  vital  operational  requirements 
should  have  an  in-house  SDA  function.  Likewise,  one-of-a-kind  systems  should  have  an  in-house 
SDA.  Medium  to  large  quantity  acquisitions  having  a  high  degree  of  rapidly  changing  technology 
that  depends  on  commercial  standards  are  normally  best  served  by  a  qualified  contractor  SDA, 
although  retention  of  the  SDA  function  by  the  TDA  is  always  appropriate  (qualified  consultative 
contractor  support  can  be  used  to  supplement  in-house  expertise). 

DESIGN/DEVELOPMENT  AGENT  (DA) 

The  Design/Development  Agent  (DA)  translates  performance  requirements  (including  those  for 
product  improvements)  into  a  product  design.  Except  for  rare  cases  in  development  acquisitions  of 
few-of-a-kind  unique  requirement  systems,  the  DA  is  a  contractor.  For  COTS,  ROTS,  etc.,  the  DA  is 
the  commercial  producer  of  the  product,  although  a  vendor  independent  of  the  producer  may  be  the 
actual  source  of  supply.  In  NDI  systems,  the  DA  is  totally  independent  of  the  Manager;  however,  the 
DA  is  the  ultimate  source  of  both  the  product  and  the  design  information  that  is  needed  for  operation, 
maintenance,  and  training  support.  The  following  NDI  issues  are  important  as  related  to  the  DA: 

•  Obtaining  interface  design  information — this  can  be  especially  difficult  when  market-driven 
standards  or  company  proprietary  standards  are  involved. 

•  Obtaining  system  assurance  information  of  sufficiently  high  confidence — this  type  of  informa¬ 
tion  may  not  be  available  or  may  be  available  in  a  form  that  is  difficult  to  interpret;  there  may 
be  insufficient  data  to  achieve  high  confidences,  especially  for  NewCOTS. 

•  Translating  DA  sourced  information  into  a  form  usable  with  the  system  or  incorporating  sup¬ 
plier  information  into  final  system  information — the  commercial  forms  and  company-specific 
formats  are  often  not  acceptable  and  may  not  be  available  in  an  electronic  form  that  makes  the 
information  easy  to  manipulate. 

•  Configuration  management  information — CM  is  not  conducted  within  the  product  but  at  the 
product  interfaces;  nevertheless,  the  DA  seldom  has  any  contractual  responsibility  to  provide 
even  notifications  of  changes. 

Since  the  DA  is  not  available  to  perform  these  functions  in  an  NDI  acquisition,  the  functions  must  be 
picked  up  by  other  agents  or  activities.  Initially,  these  functions  may  be  assumed  by  the  System  Inte¬ 
grator;  however,  all  of  the  functions  may  be  accomplished  by  the  AEA/ISEA  functions. 

ACQUISITION  ENGINEERING  AGENT  (AEA) 

The  AEA  performs  all  of  the  functions  of  the  ISEA  in  support  to  the  Manager  during  the  acquisi¬ 
tion  phase.  Usually,  the  AEA,  a  DoD  agent,  is  also  the  same  activity  that  is  planned  to  be  the  ISEA; 
this  promotes  the  smooth  transition  of  system  support  from  the  acquisition  phase  to  the  operation  and 
support  phase.  The  following  NDI  issues  are  important  to  the  AEA: 

•  Covering  the  gaps  created  by  the  inaccessibility  of  the  DA — see  DESIGN/DEVELOPMENT 
AGENT  (DA)  section  above. 

•  Conducting  or  overseeing  workmanship/quality  screens  (such  as  Electronic  Stress  Screening 
[ESS])  added  as  an  incoming  inspection  screen  or  to  a  production  line. 
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Preparing  support  plans  to  effectively  and  appropriately  use  commercial  support  capabilities. 
Providing  the  infrastructure  to  use  warranty  service,  when  implemented. 

•  Creating  an  appropriate  configuration  management  system. 

SYSTEM  INTEGRATOR/SYSTEM  INTEGRATION  AGENT  (SIA) 

The  System  Integrator/System  Integration  Agent  (SIA)  function  is  responsible  for  ensuring  com¬ 
patibility  of  all  elements  that  make  up  a  single  system  and  for  identifying/documenting  the  interface 
requirements  external  to  the  system  throughout  the  system  life.  The  SIA  may  be  either  a  DoD  activ¬ 
ity  or  a  contractor,  but  in  either  case,  the  SIA  must  maintain  a  close  liaison  with  the  other  agents.  A 
DoD  (“in-house”)  activity  is  normally  chosen  for  small-quantity  acquisitions  as  NDI  can  often  be 
purchased  and  integrated  faster  than  an  integration  contract  can  be  awarded.  Furthermore,  in-house 
SIA  assets  can  be  more  responsive  to  rapidly  changing  user  requirements  when  close  liaison  is 
needed  between  the  TDA,  SDA,  and  SIA.  Contractor  SIA  assets  are  useful  for  very-high-quantity 
acquisitions.  The  large  zone  between  these  extremes  can  be  satisfied  by  either  in-house  or  contractor 
assets  as  long  as  the  essential  tasks  are  identified  and  accomplished.  The  SIA  is  normally  responsi¬ 
ble  for  demonstrating  system  performance,  safety,  operability,  interoperability  with  interfaced/legacy 
systems,  reliability,  maintainability,  and  human  factors  performance.  The  SIA  manages  interfaces 
throughout  the  acquisition  phase,  monitoring  system  tolerances  and  error  budgets  for  all  elements  of 
the  system.  Some  or  all  of  the  SIA  responsibilities  may  be  transitioned  to  the  LCMA  for  the  opera¬ 
tion  and  support  phase.  The  following  NDI  issues  are  important  to  the  SIA: 

Interface  documentahon,  especially  second-order  parameters  (essential  interface  characteristics 
not  controlled  explicitly  by  published  interface  specifications)  and  hardware/software  interface 
requirements. 

•  Identifying,  designing,  and  executing  appropriate  modifications. 

•  Acquiring  non-off-the-shelf  classes  of  NDI. 

Determining  which  system  level  of  complexity  best  uses  NDI  and  achieves  the  most  affordable 
system  implementation  for  the  system  life  cycle.  This  includes  meeting  the  safety,  supportabil- 
ity,  and  human  factors  requirements  while  not  imposing  unreasonable  or  unaffordable  modifi¬ 
cation  requirements  on  the  NDI  candidates. 

Maintaining  system  conformity  to  changing  operational  requirements. 

•  Implementing  requirements  flowed  down  by  the  TDA  or  SDA. 

The  SIA  may  be  the  same  activity  as  the  SDA  or  LCMA  or  may  remain  an  independent  agent, 
depending  on  which  issues  are  driving  the  acquisition  and  support  decisions. 

SYSTEM  TEST  AGENT  (STA) 

The  System  Test  Agent  (STA)  supports  the  Manager  and  TDA  in  planning  and  directing  all  levels 
of  the  system  test  program.  The  STA  serves  as  a  direct  liaison  with  the  Operational  Test  and  Evalua¬ 
tion  Agency /Forces.  The  STA  acquires  access  to  test  facilities  (including  operational  forces)  needed 
for  system  tests  and  demonstrations  and  oversees  and  analyzes  results  of  tests  done  by  other  activi¬ 
ties.  The  following  NDI  issues  are  important  to  the  STA: 

•  Evaluating  contractor  conducted  tests/test  data. 

•  Determining  test  confidence  levels. 
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•  Designing  screening  tests  for  potential  NDI. 

•  Evaluating  demonstrations  as  a  part  of  source  selection. 

•  Test/conformance  requirements  within  procurement  requirements. 

The  STA  function  is  sometimes  shared  between  the  TDA  and  SIA  rather  than  being  a  separately 
identified  agent.  A  separately  identified  DoD  (“in-house”)  agent  is  recommended  for  NDI  acquisi¬ 
tions  because  of  the  specialized  expertise  needed  to  address  the  above  issues,  even  if  that  agent  is  a 
part  of  the  same  activity  as  the  TDA. 

LIFE-CYCLE  MANAGEMENT  AGENT  (LCMA) 

The  Life-Cycle  Management  Agent  (LCMA)  is  the  DoD  agent  responsible  for  the  overall  coor¬ 
dination  of  the  support  management  of  the  system  from  its  acquisition  through  its  phase-out  and  dis¬ 
posal.  This  level  of  system  management  includes  system  life-cycle  planning  and  the  coordination  of 
activities  assigned  to  the  SDA,  ISEA,  SSA,  SIA,  STA,  and  TA.  The  following  NDI  issues  are  impor¬ 
tant  to  the  LCMA: 

•  System  life-cycle  planning,  including  the  planning  impact  on  the  system  architecture  and  parti¬ 
tioning  decision  process. 

•  Specification  control  of  the  form,  fit,  and  function  of  the  system  components,  including  all 
interfaces,  across  all  installations. 

•  Advanced  planning  for  support,  including  interim  support  requirements  (which  must  take  into 
account  rapid  acquisitions  that  may  occur  in  substantially  less  time  than  normal  support  proce¬ 
dures  can  be  implemented). 

•  Configuration  management,  especially  configuration  status  accounting,  across  each  installa¬ 
tion. 

•  Life-cycle  cost  considerations  throughout  the  system  life  and  especially  in  source  selection. 

•  Characterizing  the  market(s)  supporting  the  system  both  technically  and  economically. 

•  Continuous  market  analysis/research  promoting  cost-effective  support  and  product  improve¬ 
ment  decisions. 

•  Managing  system  requirements  in  post-acquisition  phases,  including  product  improvements. 
This  is  especially  critical  for  requirements  stemming  from  the  system  being  a  part  of  a  larger 
set  of  systems  forming  an  operational  capability. 

•  Monitoring  casualty  reports,  maintenance  reports,  supply  reports,  and  other  system  metrics  of 
the  system  and  related/interfaced  systems  to  ensure  continued  operational  suitability. 

•  Establishing  and  coordinating  organizational  relationships  that  ensure  that  (rapidly  changing) 
information  is  shared  between  the  participating  support  agents.  This  includes  coordinating 
ISEA  and  SSA  support  to  users  involving  more  than  one  ISEA  and  one  SSA. 

•  Coordinating  installations  and  improvement  implementations  with  the  planning  organizations 
of  the  platform  engineering  and  repair  activities  responsible  for  maintaining  the  platform  con¬ 
figuration  management.  This  includes  participating  in  Class  Improvement  Plan  Engineering/ 
Projected  Class  Baseline  design  efforts  to  ensure  that  the  requirements  generated  are  flowed 
into  the  system  requirements  appropriately  and  realistically.  It  also  includes  the  coordination 
of  alteration  and  repair  packages  among  the  multiple  support  agents. 
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•  Ensuring  that  the  multiple  fielded  configurations  remain  interoperable  and  maintaining  plans 
to  accomplish  this.  (This  is  especially  critical  for  systems  employing  a  substantial  amount  of 
integration  software.) 

The  duties  of  the  LCMA  overlap  substantially  with  the  SDA  during  the  acquisition  phase,  so  the 
LCMA  will  normally  also  be  the  SDA  when  the  SDA  is  not  a  contractor.  The  LCMA  function 
includes  the  function  of  the  Combat  System  In-Service  Engineering  Agent  (CSISEA)  sometimes 
identified  by  Systems  Commands. 

IN-SERVICE  ENGINEERING  AGENT/ACTIVITY  (ISEA) 

The  In-Service  Engineering  Agent/ Activity  (ISEA),  a  DoD  agent,  performs  design  verification  and 
validation,  system  assurance  (especially  safety  and  quality),  documentation,  production  support,  data 
analysis,  maintenance  engineering,  installation  design  and  support,  fleet  support  of  prototype 
systems,  training  and  manning  assistance,  integrated  logistics  support  planning,  data  management, 
configuration  management,  test/support  equipment  analysis  and  support,  supply  support  planning, 
and  repair  facility  planning  and  implementation  support  to  the  Manager  throughout  the  post-acquisi¬ 
tion/operations  and  support  phase.  The  following  NDI  issues  are  important  to  the  ISEA; 

•  Configuration  management,  especially  Configuration  Status  Accounting,  of  multiple  unique 
fielded  configurations. 

•  Hardware-Software/Firmware  interfaces  and  interface  documentation/specifications. 

•  Uncontrolled  changes/product  improvements  introduced  by  product  suppliers. 

•  Implementing  engineering  changes  into  the  system. 

•  Generating  and  maintaining  accurate  system  metrics. 

•  Maintaining  qualified  sources  of  supply  for  spares/replacements. 

•  Managing  contractor  services,  including  warranty  support. 

•  Enforcing  warranty  provisions  in  the  field,  including  maintaining  accurate  warranty  data. 

SOFTWARE  SUPPORT  ACTIVITY  (SSA) 

The  Software  Support  Activity  (SSA)  performs  design  verification  and  validation,  system  assur¬ 
ance  (especially  safety  and  quality),  documentation,  production  support,  data  analysis,  maintenance 
engineering,  installation  design  and  support,  fleet  support  of  prototype  systems,  training  and  man¬ 
ning  assistance,  integrated  logistics  support  planning,  data  management,  configuration  management, 
and  software/firmware  test/support  equipment  analysis  and  support  to  the  Manager  throughout  the 
post-acquisition/operations  and  support  phase  for  system  software.  The  SSA  usually  provides  the 
planning  and  implementation  of  the  Computer  Resources  Life-Cycle  Management  Plan  as  tasked  by 
the  Manager.  The  SSA  controls  and  coordinates  the  distribution  of  software  releases  to  the  user 
community.  In  addition,  the  SSA  often  also  provides  independent  verification  and  validation  ser¬ 
vices,  independent  software  quality  assurance  services,  and/or  independent  testing  services  during 
developments,  including  modified  NDI  and  integrated  NDI.  This  promotes  the  smooth  transition 
from  the  acquisition  phase  into  the  support  phase.  The  following  NDI  issues  are  important  to  the 
SSA; 
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•  Configuration  management,  especially  Configuration  Status  Accounting,  of  multiple  unique 
fielded  configurations. 

•  Hardware-Software/Firmware  interfaces  and  interface  documentation/specifications. 

•  Uncontrolled  changes/product  improvements  introduced  by  product  suppliers. 

•  Implementing  engineering  changes  into  the  system. 

•  Generating  and  maintaining  accurate  system  metrics. 

•  Documenting  system  requirements  for  the  hardware-software  or  hardware-firmware  interface. 

•  Tailored  support  of  NDS,  including  the  requalification  of  NDS  and  modified  NDS  changes. 

•  Documenting  modifications  and  new  designs  in  modified  NDI  and  integrated  NDI. 

The  SSA  is  almost  always  a  DoD  agent;  however,  SSA  functions  have  been  successfully  contracted 
to  an  industry  agent  under  the  limited  circumstances  of  wholly  proprietary  software  that  has  been 
specially  modified  for  system  requirements.  When  the  SSA  function  is  contracted  out,  the  LCMA  or 
ISEA  should  normally  control  the  contract. 

TRAINING  AGENT/ACTIVITY  (TA) 

The  Training  Agent/Activity  (TA)  is  the  agent  for  the  cognizant  DoD  personnel  and  training  com¬ 
mand  (Chief  of  Naval  Education  &  Training  (CNET)  in  the  Navy)  for  the  training  life-cycle  support 
of  the  system.  The  TA  is  responsible  for  maintaining  current  training  for  the  system.  This  includes 
the  materials  and  curriculum  for  any  on-the-job  or  embedded  training  as  well  as  training  at  schools. 
The  TA  also  oversees  the  conduct  of  the  training  to  ensure  training  quality  and  may  also  oversee  per¬ 
sonnel  qualification  standards.  The  following  NDI  issues  are  important  to  the  TA: 

•  Maintaining  quality,  cost-effective  training. 

•  Maintaining  accurate  training  documentation. 

•  Consistency  of  system  documentation  with  the  delivered  product  (this  being  a  serious  potential 
problem  as  the  product  may  change  significantly  over  its  life). 

•  Multiple  fielded  configurations  of  the  system  with  different  operation  and  maintenance  charac¬ 
teristics  (and  potentially  unique  training  requirements). 

•  New  skill  requirements  being  introduced  by  product  improvements  (especially  those  product 
improvements  introduced  in  COTS  equipments  that  are  not  ordered  in  the  acquisition  of  the 
system  or  its  replacement  parts). 
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